Http request fails with ValueError behind https proxy, when proxy specified is an Ip Address.

[vabhasin]

Describe the bug
Boto Client configured with proxy settings fails when sending urllib3 http request behind a https proxy.
Request fails with error "ValueError: check_hostname requires server_hostname" and happens only when you specify proxy address using ip address. I was able to confirm that the same code works when you provide proxy server with a hostname instead of Ip Address.

Client creation code:

client = boto3.client('cloudwatch', 'us-east-1', config=config.Config(proxies={'https': 'https://1.2.3.4:5678'}, 
proxies_config={'proxy_ca_bundle': '/home/user/ca.pem'}))
response = client.put_metric_data(Namespace='namespace',
                                  MetricData=sample_metric)

Looking into the stack trace the error comes from ssl.py.

if context.check_hostname and not server_hostname:
            raise ValueError("check_hostname requires server_hostname")

check_hostname in the ssl_context has been set to True by botocore here.

botocore/botocore/httpsession.py

Lines 341 to 343 in 0cc6713

	# urllib3 disables this by default but we need
	# it for proper proxy tls negotiation.
	context.check_hostname = True

urllib3 doesn't pass server_hostname to _ssl_wrap_socket_impl as the server_hostname is an Ip address in our case.
This was changed in a recent fix in urllib3, according to which SNI should only be sent if the proxy is a hostname, not an Ip address.

    # If we detect server_hostname is an IP address then the SNI
    # extension should not be used according to RFC3546 Section 3.1
    use_sni_hostname = server_hostname and not is_ipaddress(server_hostname)
    # SecureTransport uses server_hostname in certificate verification.
    send_sni = (use_sni_hostname and HAS_SNI) or (
        IS_SECURETRANSPORT and server_hostname
    )
    # Do not warn the user if server_hostname is an invalid SNI hostname.
    if not HAS_SNI and use_sni_hostname:
        warnings.warn(
            "An HTTPS request has been made, but the SNI (Server Name "
            "Indication) extension to TLS is not available on this platform. "
            "This may cause the server to present an incorrect TLS "
            "certificate, which can cause validation failures. You can upgrade to "
            "a newer version of Python to solve this. For more information, see "
            "https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html"
            "#ssl-warnings",
            SNIMissingWarning,
        )

    if send_sni:
        ssl_sock = _ssl_wrap_socket_impl(
            sock, context, tls_in_tls, server_hostname=server_hostname
        )
    else:
        ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls)
    return ssl_sock

This causes the Value error when using ip address to specify proxy. According to urllib3 fix, botocore should also maybe check if the proxy settings specify ip address and set check_name accordingly.

Steps to reproduce
Setup a https proxy server. Create a boto client with proxy config specifying proxy server with Ip Address and a proxy_ca_bundle . Perform a put_metric_data call to cloudwatch

import boto3
from botocore import config

PROXY_CA_BUNDLE = "/home/user/ca.pem"
PROXIES = {'https': 'https://1.2.3.4:8889'}
client = boto3.client('cloudwatch', 'us-east-1', config=config.Config(proxies=PROXIES,
                                                                      proxies_config={'proxy_ca_bundle': PROXY_CA_BUNDLE}))
response = client.put_metric_data(Namespace='namespace',
                                  MetricData=sample_metric)
print(response)

Expected behavior
Request should succeed even if proxy is specified with an ip address.

Debug logs
stacktrace boto3 proxy.txt

Package versions
Botocore: 1.24.25
Boto3: 1.21.23
urllib3: 1.26.9

[tim-finnigan]

Thanks @vabhasin for reporting, it looks like this will require a fix just as urllib3 did.

[nateprewitt]

Hi @vabhasin, the change was released today. Would you mind confirming you're no longer experiencing this issue? Thanks! At that point, we should be set to resolve this.

[vabhasin]

@nateprewitt Sorry for the late response. I am no longer experiencing the issue. Thank you for the help!

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.