TLS-in-TLS not working with IP proxies #2400

janemba · 2021-09-01T11:44:41Z

Subject

I'm trying to send a GET HTTP request to an HTTPS website through a secure proxy. The proxy only allows TLS connection with basic authentication.

To achieve this, I use a ProxyManager.

As result the following exceptions is triggered:

ValueError: check_hostname requires server_hostname

Environment

>>> print("OS", platform.platform())
OS Linux-5.12.13-x86_64-Intel-R-_Core-TM-_i7-8750H_CPU_@_2.20GHz-with-glibc2.33
>>> print("Python", platform.python_version())
Python 3.9.6
>>> print("urllib3", urllib3.__version__)
urllib3 1.25.11

Steps to Reproduce

You need an HTTPS proxy with basic authentication and an HTTPS website as an example:

def get(url, proxy, username, password):

    headers  = urllib3.make_headers(proxy_basic_auth=f'{username}:{password}')
    proxy    = urllib3.ProxyManager('https://{}'.format(proxy), proxy_headers=headers)
    response = proxy.request("GET", url)            # Exception trigger here

    return (response)

Expected Behavior

Website body page as a response.

Actual Behavior

Exception triggered:

Traceback (most recent call last):
[...]
  File "/home/janemba/.virtualenv/test-rCp672mu-py3.9/lib64/python3.9/site-packages/urllib3/request.py", line 74, in request
    return self.request_encode_url(
  File "/home/janemba/.virtualenv/test-rCp672mu-py3.9/lib64/python3.9/site-packages/urllib3/request.py", line 96, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "/home/janemba/.virtualenv/test-rCp672mu-py3.9/lib64/python3.9/site-packages/urllib3/poolmanager.py", line 532, in urlopen
    return super(ProxyManager, self).urlopen(method, url, redirect=redirect, **kw)
  File "/home/janemba/.virtualenv/test-rCp672mu-py3.9/lib64/python3.9/site-packages/urllib3/poolmanager.py", line 375, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/home/janemba/.virtualenv/test-rCp672mu-py3.9/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 696, in urlopen
    self._prepare_proxy(conn)
  File "/home/janemba/.virtualenv/test-rCp672mu-py3.9/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 964, in _prepare_proxy
    conn.connect()
  File "/home/janemba/.virtualenv/test-rCp672mu-py3.9/lib64/python3.9/site-packages/urllib3/connection.py", line 359, in connect
    conn = self._connect_tls_proxy(hostname, conn)
  File "/home/janemba/.virtualenv/test-rCp672mu-py3.9/lib64/python3.9/site-packages/urllib3/connection.py", line 500, in _connect_tls_proxy
    return ssl_wrap_socket(
  File "/home/janemba/.virtualenv/test-rCp672mu-py3.9/lib64/python3.9/site-packages/urllib3/util/ssl_.py", line 453, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls)
  File "/home/janemba/.virtualenv/test-rCp672mu-py3.9/lib64/python3.9/site-packages/urllib3/util/ssl_.py", line 495, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock)
  File "/usr/lib64/python3.9/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/lib64/python3.9/ssl.py", line 997, in _create
    raise ValueError("check_hostname requires server_hostname")
ValueError: check_hostname requires server_hostname

The text was updated successfully, but these errors were encountered:

sethmlarson · 2021-09-01T11:51:28Z

TLS-in-TLS proxy support was added in urllib3 v1.26.0, could you upgrade and try again?

janemba · 2021-09-01T12:08:11Z

I tried with version 1.26.6 and I got the same result.

Traceback (most recent call last):
  File "/home/janemba/projects/test/examples/urllibway.py", line 51, in <module>
    main(opt.url)
  File "/home/janemba/projects/test/examples/urllibway.py", line 31, in main
    response = get(url, proxy, login, password)
  File "/home/janemba/projects/test/examples/urllibway.py", line 15, in get
    response = proxy.request("GET", url)
  File "/home/janemba/.virtualenv/test-rCp672mu-py3.9/lib64/python3.9/site-packages/urllib3/request.py", line 74, in request
    return self.request_encode_url(
  File "/home/janemba/.virtualenv/test-rCp672mu-py3.9/lib64/python3.9/site-packages/urllib3/request.py", line 96, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "/home/janemba/.virtualenv/test-rCp672mu-py3.9/lib64/python3.9/site-packages/urllib3/poolmanager.py", line 532, in urlopen
    return super(ProxyManager, self).urlopen(method, url, redirect=redirect, **kw)
  File "/home/janemba/.virtualenv/test-rCp672mu-py3.9/lib64/python3.9/site-packages/urllib3/poolmanager.py", line 375, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/home/janemba/.virtualenv/test-rCp672mu-py3.9/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 696, in urlopen
    self._prepare_proxy(conn)
  File "/home/janemba/.virtualenv/test-rCp672mu-py3.9/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 964, in _prepare_proxy
    conn.connect()
  File "/home/janemba/.virtualenv/test-rCp672mu-py3.9/lib64/python3.9/site-packages/urllib3/connection.py", line 359, in connect
    conn = self._connect_tls_proxy(hostname, conn)
  File "/home/janemba/.virtualenv/test-rCp672mu-py3.9/lib64/python3.9/site-packages/urllib3/connection.py", line 500, in _connect_tls_proxy
    return ssl_wrap_socket(
  File "/home/janemba/.virtualenv/test-rCp672mu-py3.9/lib64/python3.9/site-packages/urllib3/util/ssl_.py", line 453, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls)
  File "/home/janemba/.virtualenv/test-rCp672mu-py3.9/lib64/python3.9/site-packages/urllib3/util/ssl_.py", line 495, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock)
  File "/usr/lib64/python3.9/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/lib64/python3.9/ssl.py", line 997, in _create
    raise ValueError("check_hostname requires server_hostname")
ValueError: check_hostname requires server_hostname

$ poetry show
[...]
urllib3            1.26.6    HTTP library with thread-safe connection pooling, file post, and more.

janemba · 2021-09-01T12:19:47Z

Is there any way to get some log related to the TLS certificate processing from urllib3 ?

pquentin · 2021-09-01T12:22:45Z

Are you connecting to your proxy using an IP address?

janemba · 2021-09-01T13:04:12Z

Are you connecting to your proxy using an IP address?

Yes, I'm using the following format https://1.2.3.4:443

pquentin · 2021-09-01T14:38:55Z

Congratulations, you just found a bug! We send SNI even though we should not when the proxy is an IP. (That's what the error message says.)

A workaround would be to use an hostname if you have one or just stop using TLS-in-TLS. (This has nothing to do with basic authentication.)

janemba · 2021-09-01T15:24:46Z

Oh thank you :-(

Well I will wait for an update :-D

Kcam9908 · 2021-09-07T12:30:22Z

Hi there, I'm currently working on a project that relies on a vulnerability fixed in v1.26.5, but this issue keeps me from using our proxies correctly. I have a deadline to meet this week, and was wondering if there is an estimated release that fixes this bug, or possibly a workaround?

sethmlarson · 2021-09-07T15:22:35Z

@Kcam9908 TLS-in-TLS proxies were added in 1.26.0 and if I'm understanding correctly this issue has always been in the TLS-in-TLS proxies. You should be able to use HTTPS-to-HTTP and HTTP-to-HTTPS proxies which have "always" been available in urllib3 just fine. What I'm saying is you probably aren't "newly" impacted by this issue if you've recently upgraded to 1.26.x?

Does this make sense? Let me know if I forgot something.

Kcam9908 · 2021-09-07T15:43:39Z

No, it is not a new problem, version-wise. All current 1.26.x versions are not working. My team was using an out of date version (1.25.8), and until recently we didn't have to update. We found a vulnerability that required updating, which then broke it.

sethmlarson · 2021-09-07T15:50:23Z

Could you share your configuration? There was an unfortunate situation in urllib3 where we never documented support for TLS-in-TLS proxies pre-1.26.0 but trying to use it didn't fail with an error it instead "kinda worked" by doing HTTP on the proxy and HTTPS for the target.

Basically your "solution" may be to use a proxy with http:// and target https:// for the website you're visiting which was the pre-1.26.0 behavior you were likely experiencing anyways.

See #1850 for another explanation, you would have seen this warning if you were using 1.25.9 which is the version that added the warning in preparation for us actually supporting TLS-in-TLS proxying.

Kcam9908 · 2021-09-10T14:38:11Z

I noticed there was a PR opened. Do you guys have a timeline for when the fix for this issue will be released? My team and I are trying to make a decision moving forward, as it is a critical fix for our product.

sethmlarson · 2021-09-10T14:41:08Z

@Kcam9908 Did you read my reply to you directly above? In summary we don't think the issue you're seeing will be resolved by the PR you mentioned.

"as it is a critical fix for our product."

Your organization might consider sponsoring our development efforts if urllib3 is critical to your business.

Kcam9908 · 2021-09-10T15:02:09Z

Thanks for the response, I will look into the workaround. For management purposes, could you please let me know when this fix will be released?

sethmlarson · 2021-09-10T15:04:24Z

@Kcam9908 There is no set timeline, we'll start the release process when the PR is merged.

sethmlarson · 2021-09-16T13:08:55Z

The PR for relying on urllib3's hostname matching logic for verifying proxy certificates is merged. However there's still a bug with IPv6 addresses we'd like to fix before we ship a bugfix release.

pquentin · 2021-09-24T09:23:28Z

This was fixed in #2419 and released as part of 1.26.7.

pquentin changed the title ~~Secure proxy (TLS-in-TLS) with basic authentication not working~~ TLS-in-TLS not working with IP proxies Sep 1, 2021

janemba closed this as completed Sep 1, 2021

pquentin reopened this Sep 1, 2021

jalopezsilva self-assigned this Sep 1, 2021

opsdisk mentioned this issue Sep 3, 2021

Add option for self-signed certs opsdisk/yagooglesearch#3

Closed

jalopezsilva mentioned this issue Sep 8, 2021

Rely on urllib3 hostname matching for HTTPS proxy validation. #2407

Merged

pquentin mentioned this issue Sep 9, 2021

Hostname matching doesn't strip trailing '.' when connecting with proxy #2244

Closed

pquentin closed this as completed Sep 24, 2021

vabhasin mentioned this issue Mar 24, 2022

Http request fails with ValueError behind https proxy, when proxy specified is an Ip Address. boto/botocore#2645

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TLS-in-TLS not working with IP proxies #2400

TLS-in-TLS not working with IP proxies #2400

janemba commented Sep 1, 2021

sethmlarson commented Sep 1, 2021

janemba commented Sep 1, 2021 •

edited

janemba commented Sep 1, 2021

pquentin commented Sep 1, 2021 •

edited

janemba commented Sep 1, 2021

pquentin commented Sep 1, 2021

janemba commented Sep 1, 2021

Kcam9908 commented Sep 7, 2021

sethmlarson commented Sep 7, 2021

Kcam9908 commented Sep 7, 2021

sethmlarson commented Sep 7, 2021

Kcam9908 commented Sep 10, 2021 •

edited

sethmlarson commented Sep 10, 2021 •

edited

Kcam9908 commented Sep 10, 2021

sethmlarson commented Sep 10, 2021

sethmlarson commented Sep 16, 2021

pquentin commented Sep 24, 2021

TLS-in-TLS not working with IP proxies #2400

TLS-in-TLS not working with IP proxies #2400

Comments

janemba commented Sep 1, 2021

Subject

Environment

Steps to Reproduce

Expected Behavior

Actual Behavior

sethmlarson commented Sep 1, 2021

janemba commented Sep 1, 2021 • edited

janemba commented Sep 1, 2021

pquentin commented Sep 1, 2021 • edited

janemba commented Sep 1, 2021

pquentin commented Sep 1, 2021

janemba commented Sep 1, 2021

Kcam9908 commented Sep 7, 2021

sethmlarson commented Sep 7, 2021

Kcam9908 commented Sep 7, 2021

sethmlarson commented Sep 7, 2021

Kcam9908 commented Sep 10, 2021 • edited

sethmlarson commented Sep 10, 2021 • edited

Kcam9908 commented Sep 10, 2021

sethmlarson commented Sep 10, 2021

sethmlarson commented Sep 16, 2021

pquentin commented Sep 24, 2021

janemba commented Sep 1, 2021 •

edited

pquentin commented Sep 1, 2021 •

edited

Kcam9908 commented Sep 10, 2021 •

edited

sethmlarson commented Sep 10, 2021 •

edited