Vault +v1.9 - no_proxy is ignored

[anthosz]

Describe the bug
NO_PROXY env doesn't works anymore since upgrade from v1.8 to v1.9.4

To Reproduce
Steps to reproduce the behavior:

1. Run export VAULT_API_ADDR="https://vault:8200" VAULT_CLUSTER_ADDR="https://vault:8201" ETCD_ADDR="https://etcd1:2379" HTTPS_PROXY="http://PROXY:PORT/" NO_PROXY="etcd1"; vault server -config /vault
2. no_proxy seems not to works anymore

"error":"rpc e
rror: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"t[.... proxy output that tell us that it's internal resource so cannot works....]
2022-04-13T18:58:48.426+0200 [INFO]  proxy environment: https_proxy=http://PROXY:PORT/ no_proxy="etcd1"

Expected behavior
I expect that proxy is ignored for etcd1 server.

Environment:

• Vault Server Version (retrieve with vault status): 1.9.4
• Vault CLI Version (retrieve with vault version): 1.9.4

Additional context
If I rollback to v1.8, all works.
With Vault 1.9, it seems to force the proxy and to ignore the no_proxy env even it's set in the logs.
Seems related to PR #12582
I also tried to unset/set VAULT_HTTP_PROXY / no_proxy / NO_PROXY but always the same issue.

[peteski22]

Hi anthosz, thanks for logging this issue, I'm going to take a look and see if I can help out.

Would it be possible for you to help me figure out a robust way to easily reproduce this?

I see you have some steps in the initial report, thanks for that, but I was wondering if maybe we would be able to come up with a set of steps that allowed us to reproduce it with Vault running in dev server mode?

What set up would we need and what would trigger the error you're seeing?

With regards to the environment variables

You are setting:

• HTTPS_PROXY
• NO_PROXY
• VAULT_API_ADDR
• VAULT_CLUSTER_ADDR
• ETC_ADDR

You are not setting:

• VAULT_HTTP_PROXY

The reason I ask about not setting VAULT_HTTP_PROXY is that the PR you referenced replaces the behaviour of proxy resolution only when that environment variable is set.

The last thing I wanted to check was:

I also tried to unset/set VAULT_HTTP_PROXY / no_proxy / NO_PROXY but always the same issue.

So just to double check, when you run Vault setting VAULT_HTTP_PROXY you are performing the following two additional use cases:

1. Setting VAULT_HTTP_PROXY and setting NO_PROXY
2. Unsetting VAULT_HTTP_PROXY and unsetting NO_PROXY

Thanks again for reporting this and for your help.

[anthosz]

Hi @peteski22,

I just try few matrix with the last version of Vault (issue started on v1.9.0) and at the end it works:

For the example, let's use:
export NO_PROXY=example.org HTTPS_PROXY=http://proxy:port/

Matrix:

• NO_PROXY + HTTPS_PROXY = doesn't works -> proxy forced for example.org also
• VAULT_HTTP_PROXY + NO_PROXY + HTTPS_PROXY = doesn't works -> proxy forced for example.org also
• VAULT_HTTP_PROXY + NO_PROXY = works -> proxy bypassed for example.org

So for me, the issue is "fixed" but it didn't works as expected or maybe the doc need to be updated (as far as I see, it was supposed to be transparent for existing env)?

Thank you!

[peteski22]

Thanks for getting back @anthosz.

Hmmn. Whilst I am glad it seems to be fixed for you, let's see if we can figure out what would make things more clear in the docs?

I'm still curious about the environment variables set in each of the three items in your matrix, could you provide an example like this?

Case	NO_PROXY	HTTP_PROXY	HTTPS_PROXY	VAULT_HTTP_PROXY
1	example.org		http://proxy:port
2	example.org		http://proxy:port	?
3	example.org			?

My understanding of the default behaviour (ProxyFunc) in http/httpproxy/proxy.go is as follows:

If the request is HTTP then the underlying Go package attempts to resolve a parsed URL from the HTTP_PROXY environment variable (this will be nil if the environment variable wasn't set or cannot be parsed as a URL). Similarly, if the request is HTTPS then it attempts to use the parsed URL from the HTTPS_PROXY environment variable (again this will be nil if the environment variable wasn't set or couldn't be parsed as a URL).

After this, if the selected (HTTP/HTTPS) proxy URL is nil then the proxy is ignored (as we have no proxy to use even if the request address doesn't match something in NO_PROXY).

If the selected proxy URL is present, the request address is passed to useProxy which uses the values configured in NO_PROXY/no_proxy to determine if any matches are found. If they are the proxy can be bypassed, so nil is returned, if not, the URL of the relevant proxy is returned (as covered above).

This doesn't seem to line up with what you're experiencing.

Furthermore, when supplying VAULT_HTTP_PROXY this behaviour is overridden entirely and all requests are supplied with the Vault proxy value. This is what has been configured in the pull request you mentioned when logging the issue:

https://github.com/hashicorp/vault/pull/12582/files#diff-aa9bfd1a638fbb706f8e8920297902937011160319d9679add5dca56e5ab8277R391

We could update the documentation to make it very clear that supplying VAULT_HTTP_PROXY will change the behaviour for all Vault requests (that they will ignore NO_PROXY), would that link up with what you're seeing? I'm not sure I'm getting that from the results you posted.

Thanks again for taking the time to post this bug, and to reply to my questions.

[anthosz]

@peteski22
Concerning the full matrix (and of course, I try to bypass and use the proxy for https requests):

Case	NO_PROXY	HTTP_PROXY	HTTPS_PROXY	VAULT_HTTP_PROXY	NO_PROXY status after test
1	example.org	unset	http://proxy:port	unset	ignored, it try to use PROXY to connect to example.org 👎
2	example.org	unset	http://proxy:port	http://proxy:port	ignored, it try to use PROXY to connect to example.org 👎
3	example.org	unset	unset	http://proxy:port	taken into account, it didn't use PROXY to connect to example.org 👍

I hope it's more clear now :)

Don't hesitate if you need more informations.

[anthosz]

little update few weeks later..

New deploy was done:
NO_PROXY set = example.org
VAULT_HTTP_PROXY set = http://proxy:port

If we back to the previous table:

• case 1 & 2 : it force the proxy for the NO_PROXY value (so unable to contact example.org).
• case 3: it bypass the proxy for the VAULT_HTTP_PROXY value (so unable to contact resources outside internal network)

I have the feeling that I mixed few versions during my last tests..

To summarize, unable to use Vault between v1.9.x & v1.10.x. v1.8.x is ok.

[aphorise]

I'm a little late to the party and am trying to understand as well as relate to what's next?

My understanding is that:

1. Vault specific environment variables VAULT_HTTP_PROXY always takes precedence above all other OS / configuration parameters.
2. The documentation has been update to call out OS vs VAULT_HTTP_PROXY:

   HTTP or HTTPS proxy location which should be used by all requests to access Vault. When present, this overrides the default proxy resolution behavior. Format should be http://server:port or https://server:port.

Hey @anthosz I was wondering:

• is this still applicable for you?
• have you retested in recent 1.11.3 version of Vault?
• with exactly the same ENV settings going back to 1.8.12 from 1.10.6 and or 1.11.3 all works as you expect?
• is the issue related to example.com with actual DNS entries or something internally like etcd via /etc/resolv.conf?

[anthosz]

Hi @aphorise ,

• Yes, I followed the last table to be able to use Vault with proxy (using the last version 1.11.x), new issue is that it was also mandatory to add the IPs in the proxy exception even if I use hostname in Vault....
• Yes
• yes but didn't retried since last tentative in May
• Something internally like etcd

[peteski22]

Linking: #15377 just for reference. (better late than never).