Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Streamed request with preset Host header sends two Host headers #4392

Closed
mjpieters opened this issue Nov 20, 2017 · 8 comments
Closed

Streamed request with preset Host header sends two Host headers #4392

mjpieters opened this issue Nov 20, 2017 · 8 comments
Labels

Comments

@mjpieters
Copy link
Contributor

The following code sends two host headers:

import requests

def gen(): yield b'binary data'

requests.post(
    'http://localhost:3000/', 
    data=gen(),
    headers={'Host': 'example.com'}
)

(I tested with nc -vvl 3000 on OS X).

Headers received:

POST / HTTP/1.1
Host: localhost:3000
User-Agent: python-requests/2.18.1
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
Host: example.com
Transfer-Encoding: chunked

This is because when chunking a request, we don't tell the httplib / http.client library to skip the host header if one is already set. From the adapters.HTTPAdapter.send() implementation:

                    low_conn.putrequest(request.method,
                                        url,
                                        skip_accept_encoding=True)

which leaves the skip_host parameter set to False. It probably should use skip_host='host' in request.headers here.

System Information

>>> requests.help.main()
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": "2.0.3"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.6.3"
  },
  "platform": {
    "release": "16.7.0",
    "system": "Darwin"
  },
  "pyOpenSSL": {
    "openssl_version": "1010006f",
    "version": "17.3.0"
  },
  "requests": {
    "version": "2.18.1"
  },
  "system_ssl": {
    "version": "100020cf"
  },
  "urllib3": {
    "version": "1.22"
  },
  "using_pyopenssl": true
}
@nateprewitt
Copy link
Member

Yep, this looks like a miss from #3637. Since we're not using urllib3's chunked request interface, we never got the change integrated in Requests.

The quick path here is to carry the host dedupe logic from urrlib3 into that try block. The better but likely more involved solution would be to try to use urllib3's chunked logic in Requests. This is something we've discussed before but probably out of scope for this issue.

@requests/core I'm assuming the former option is the advisable one, but do you have different thoughts?

@Lukasa
Copy link
Member

Lukasa commented Nov 20, 2017

I think we can probably use urllib3’s implementation: it’s present in all our supported versions of urllib3 I think.

@sigmavirus24
Copy link
Contributor

If we're going to swap in urllib3's chunked logic, we need better end-to-end (socket level) tests of our chunking behaviour to make sure we don't drastically change things.

@jlherren
Copy link

Does anyone know a work-around to avoid this double host header?

I need to access a frustrating and honestly quite stupid API that I have no control over, which requires the host header to be some kind of information totally unrelated to the actual hostname. And it requires chunked encoding.

@mjpieters
Copy link
Contributor Author

Currently the best work-around you have is to not use requests and use urllib3 directly.

@judofyr
Copy link

judofyr commented Apr 3, 2020

Any update on this?

@nateprewitt
Copy link
Member

Linking #5391 back here as a potential fix as converting to urllib3's chunked functionality is unlikely to happen before a major version bump.

@nateprewitt nateprewitt added the Bug label Sep 4, 2021
@sethmlarson
Copy link
Member

Requests is now using urllib3's chunking API

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

7 participants