Proxy won't fallback to IPv4 when IPv6 is not available

[amit7itz]

The issue

I encountered an issue when building one of my Dockerfiles on a host that uses the proxy (version 0.6.4).
The Dockerfile tries to run curl "https://www.random.org/cgi-bin/randbyte?nbytes=10&format=h", but half of the times it gets curl: (56) Received HTTP code 502 from proxy after CONNECT from the proxy server.

After some investigation, I found out that the DNS address www.random.org leads to 4 IP addresses, 2 are IPv4 and 2 are IPv6.

Name:   www.random.org
Address: 104.20.44.7
Name:   www.random.org
Address: 104.20.45.7
Name:   www.random.org
Address: 2606:4700:10::6814:2c07
Name:   www.random.org
Address: 2606:4700:10::6814:2d07

When it failed, this was the log in the proxy:

2021/08/17 13:13:47 [error] 83#83: *22 proxy_connect: upstream connect failed (101: Network unreachable) while connecting to upstream, client: 172.17.0.1, server: proxy_director_, request: "CONNECT www.random.org:443 HTTP/1.1", host: "www.random.org:443"

It seems like the error happens when the proxy tried to access one of the IPv6 addresses, which makes sense because my host only has an IPv4 address.

I expected that when the IPv6 leads to "Network unreachable", the Nginx will fall back to the IPv4 and run the query (just like the normal behavior of the curl command), but instead is just returns "502 Bad Gateway".

Is it possible to make it happen?
If not, maybe at least add an environment variable that controls the usage of IPv6, so it will be easier to disable?

Steps to reproduce

1. Get a Linux machine with IPv4 connectivity only.
2. On the machine, run the proxy
   sudo docker run -d --name docker_registry_proxy -p 0.0.0.0:3128:3128 rpardini/docker-registry-proxy:0.6.4
3. On the host, run multiple times curl --proxy http://localhost:3128 "https://www.random.org/cgi-bin/randbyte?nbytes=10&format=h"
4. The curl command will fail with curl: (56) Received HTTP code 502 from proxy after CONNECT about half of the times

Expected behavior

The request should be successful all the time, just like when the proxy is not used (curl "https://www.random.org/cgi-bin/randbyte?nbytes=10&format=h")

Workaround

After exploring the project's code, I managed to disable the usage of IPv6 by creating a file named resolvers.conf that contains

resolver 8.8.8.8 ipv6=off;

And mount it to the container with -v /home/ubuntu/resolvers.conf:/etc/nginx/resolvers.conf
This makes the Nginx not resolve DNS records to IPv6 addresses at all.

Thanks in advance!

[rpardini]

Hello, as you figured we do very little except point nginx to the default resolvers if not specified.
You've found the way to specify your own resolver config, which seems the correct way to workaround.

Otherwise poke around Docker, if the machine is IPv4, why is it returning AAAA records?
Also: the CONNECT proxy layer (which is the layer involved in your example) might be even more confusing than nginx's upstream resolving, since it's an external nginx module.
The fact the proxy is being used for building a Dockerfile (and affecting curl) is also version and platform dependent I remember correctly. Try docker buildx.

[amit7itz]

Hi rpardini! and thanks for the quick response 😄

I don't think it has anything to do with docker itself, because as much as I understand, Nginx has its own implementation of the DNS query. https://stackoverflow.com/a/40331256/10574201
At least in my use case, the container didn't use the docker's DNS server at any point, but an external one.

If the behavior is really caused by Nginx internal implementation and cannot be changed, maybe it's a good idea to add some env var to control the usage of IPv6, or add to the documentation how to use a custom resolver to control it?

[rpardini]

Yes. I'm hoping future versions of nginx will be more standard regarding 1) name resolutions and 2) ipv6 port binding, which requires special syntax. Feel free to PR docs as well.

[iamkhalidbashir]

This worked like a charm
It is very much need for gcr.io as it was resolving ipv6 and failing