request.get crashes with "The header content contains invalid characters" #2120

danielgindi · 2016-03-08T08:29:20Z

Try this:

require('request').get('http://www.test.com/אבגד.pdf');

After a few milliseconds this crashes with TypeError('The header content contains invalid characters')

Now if I wrap it in encodeURI - it succeeds.
But if we receive urls from an external source - they may or may not be encoded. Is there some kind of built in mechanism to deal with this? (Other than doing encodeURI(decodeURI(url)))

The text was updated successfully, but these errors were encountered:

danielgindi · 2016-03-08T08:31:53Z

Also - I found no way to catch that exception. No matter if there's a try-catch, or .on('error', ...

luiselizondo · 2016-03-09T21:02:24Z

I'm getting the same error using Node version 0.12.12 and greater. The same code works with 0.12.9:

_http_outgoing.js:355
      throw new TypeError('The header content contains invalid characters');
            ^
TypeError: The header content contains invalid characters
    at ClientRequest.OutgoingMessage.setHeader (_http_outgoing.js:355:13)
    at new ClientRequest (_http_client.js:101:14)
    at Object.exports.request (http.js:49:10)
    at Object.exports.request (https.js:136:15)
    at Request.start (//application/node_modules/request/request.js:747:30)
    at Request.end (/application/node_modules/request/request.js:1381:10)
    at end (/application/node_modules/request/request.js:575:14)
    at Immediate._onImmediate (/application/node_modules/request/request.js:589:7)
    at processImmediate [as _immediateCallback] (timers.js:367:17)

luiselizondo · 2016-03-09T21:25:31Z

Turns out that in my case I was sending a line break on a cookie. I had to remove it.

danielgindi · 2016-03-09T22:28:05Z

No, my case is a simple one-liner, no cookies, no additional headers.

pierreozoux · 2016-03-10T10:46:40Z

I get the same issue , using https://github.com/matt-major/do-wrapper when trying to access my account information.

_http_outgoing.js:348
    throw new TypeError('The header content contains invalid characters');
    ^

TypeError: The header content contains invalid characters
    at ClientRequest.OutgoingMessage.setHeader (_http_outgoing.js:348:11)
    at new ClientRequest (_http_client.js:85:14)
    at Object.exports.request (http.js:31:10)
    at Object.exports.request (https.js:197:15)
    at Request.start (/home/pierre/Documents/freelance/upwork/dosh/node_modules/do-wrapper/node_modules/request/request.js:746:30)
    at Request.write (/home/pierre/Documents/freelance/upwork/dosh/node_modules/do-wrapper/node_modules/request/request.js:1345:10)
    at end (/home/pierre/Documents/freelance/upwork/dosh/node_modules/do-wrapper/node_modules/request/request.js:560:16)
    at Immediate._onImmediate (/home/pierre/Documents/freelance/upwork/dosh/node_modules/do-wrapper/node_modules/request/request.js:588:7)
    at processImmediate [as _immediateCallback] (timers.js:383:17)

TimBeyer · 2016-03-24T10:43:19Z

Thing is, while it's easy to fix this, request should catch the error and propagate it through the error handlers. Currently it's thrown as an uncaught exception which can cause pretty bad server crashes that you can never recover from because you cannot manually try/catch this.

So I'd say this is a pretty severe bug in request. The error needs to be propagated.

TimBeyer · 2016-03-24T14:44:11Z

I've invested a good 2h now trying to fix this issue, for which writing a reproducing test case was pretty trivial.
I've come to the conclusion that with my current knowledge of the codebase it's unfixable unless the validation would be added to https://github.com/request/caseless, which btw cost me a solid 20-30 minutes to discover is actually self.setHeader. That behavior is neither documented on caseless, nor is it obvious from the name that it deals with setting and getting headers, and nor is it expected that caseless.httpify(self, self.headers) would monkey patch the object with setHeader.

try/catch at the point where the request object is created is pretty much impossible to recover from because there is no simple way to abort execution then. self.req.end will still be called, and if the call is removed none of the subsequent behavior is triggered (again, for entirely non obvious reasons).

To be blunt, it scares me that my production services rely on a library with a 1.4k line main file in which the logic is so convoluted that something as easy as propagating an error from a try/catch is practically impossible without deeply studying the entire rube goldberg machine of events. Time to move away from this.

jankronquist · 2016-04-11T11:27:27Z

This also affects Node 5.6.0 and above which has more strict header validation: https://github.com/nodejs/node/blob/v5.6.0/CHANGELOG.md

simov · 2016-04-11T12:50:14Z

Fixed here #2164

Take a look at the test on how to handle the error in your code.

danielgindi · 2016-04-11T12:57:49Z

Seems like handling the error is as usual, with an .on('error', ... event. Good!

This will make the request more bulletproof, without crashing the app on stupid things in the future.

TimBeyer · 2016-04-11T13:02:32Z

Well, now I feel like an incredibly silly prick.
I actually had exactly the same solution ready and my tests would constantly fail.
Turns out in my 'trivial test case' I forgot to call t.end().

My apologies, next time I'll just publish my branch and ask for feedback.

simov · 2016-04-12T13:10:43Z

@TimBeyer no problem, version 2.71 is published with the fix 👍

khurrumqureshi · 2016-04-12T15:09:35Z

@simov The version 2.71 has fixed the problem partially for me, Here is my scenario I am sending a post request and getting data as chunks,

request({
              method: 'POST',
              headers: {
                 'test': 'אבגד'
              },
              uri: apiUrl,
              qs: req.query,
              json: req.body,
              gzip: true
            }
            , function(error, response, body) {
               console.log('callback')
            })
            .on('response', function (response) {
             console.log(response)
            })
            .on('error', function (err) {
              console.log(err);
            });

When I run this I get an error

return self.req.write.apply(self.req, arguments)
                 ^

TypeError: Cannot read property 'write' of undefined
    at Request.write (/Users/muhammadkhurrumqureshi/Workspace/www/node_modules/request/request.js:1385:18)
    at end (/Users/muhammadkhurrumqureshi/Workspace/www/node_modules/request/request.js:565:18)
    at Immediate._onImmediate (/Users/muhammadkhurrumqureshi/Workspace/www/node_modules/request/request.js:594:7)
    at processImmediate [as _immediateCallback] (timers.js:383:17)

Can you kindly help on this?

simov · 2016-04-12T16:29:43Z

@khurrumqureshi fixed here #2165 👍

Basically the same check as in the end method. The reason why is because start is being called on the previous line, and up until now it was not expected to throw. When your request have a body write is called before end, hence the error.

khurrumqureshi · 2016-04-13T06:00:55Z

@simov 👍

bitliner · 2016-11-08T13:20:13Z

Same here.

TypeError: The header content contains invalid characters
    at ClientRequest.OutgoingMessage.setHeader (_http_outgoing.js:348:11)
    at new ClientRequest (_http_client.js:85:14)
    at Object.exports.request (http.js:31:10)
    at Object.exports.request (https.js:199:15)
    at Request.start (/home/mymodule/node_modules/request/request.js:744:32)
    at Request.end (/home/mymodule/node_modules/request/request.js:1433:10)
    at end (/home/mymodule/node_modules/request/request.js:566:14)
    at Immediate.<anonymous> (/home/mymodule/node_modules/request/request.js:580:7)
    at runCallback (timers.js:570:20)
    at tryOnImmediate (timers.js:550:5)

Response headers are:

headers:
      { 'x-backside-transport': 'OK OK,FAIL FAIL',
        connection: 'close',
        'transfer-encoding': 'chunked',
        'x-dp-local-file': 'true',
        'x-client-ip': '127.0.0.1,176.31.126.162',
        'x-global-transaction-id': '145630106',
        date: 'Tue, 08 Nov 2016 01:03:59 GMT',
        'www-authenticate': 'Basic realm="Gateway(Log-in)"',
        'x-archived-client-ip': '127.0.0.1',
        'content-type': '',
        'x-dp-tran-id': 'gateway' },
     rawHeaders:
      [ 'X-Backside-Transport',
        'OK OK,FAIL FAIL',
        'Connection',
        'close',
        'Transfer-Encoding',
        'chunked',
        'x-dp-local-file',
        'true',
        'X-Client-IP',
        '127.0.0.1,176.31.126.162',
        'X-Global-Transaction-ID',
        '145630106',
        'Date',
        'Tue, 08 Nov 2016 01:03:59 GMT',
        'WWW-Authenticate',
        'Basic realm="Gateway(Log-in)"',
        'X-Archived-Client-IP',
        '127.0.0.1',
        'Content-Type',
        '',
        'X-DP-Watson-Tran-ID',
        'gateway' ]

Headers of request are:

{ accept: 'application/json',
     authorization: 'Basic ...g==',
     referer: 'https://hostname.com/classify?text=%20So%20happy%20with%20this%20👍👍👍👍👍',
     host: 'gateway.myclass.net' }

As you can see the request contains 3 times a special character <U+1F44D>.

Might it be that the problem?

It looks like node.js https module does not support it.

henhal · 2017-01-09T18:11:51Z

Just in case someone else had the same issue as I did - I had this error with the aws-sdk and it was driving me crazy for hours. Turned out to be a PrvScan character \u001b[5~ that for some reason was lurking about in my ~/.aws/credentials file, which propagated into the Authorization header used by AWS requests. The error only appeared on my Windows 7 machine, so it was probably something with how the credentials file was retrieved from my AWS account and some Windows character encoding. It wasn't visible when cating the credentials file either, but became visible once I opened the file with vi.

lloyddugmore · 2018-03-22T22:01:23Z

Yeah, just had the same issue. Took me a few hours to figure it out I must admit.
was using MINGW64 (Git Bash) on a windows machine. Tried running the same script using Node in command prompt.... and hello.... it worked????

nkostadinov · 2018-11-20T15:03:50Z

I just hit the same error and cannot find a way to catch it. I'm building an image proxy so I cannot really controll what the other server is returning. How can I avoid carshing the whole server in production ?!

LeszekChojnacki · 2020-01-04T14:39:41Z

I had the same issue, it started to work when i exchange the "set" for "auth". For example:

it('ERROR, Wrong GET request', (done)=>{
    request(server).get("/api/")
    .set('Authorization', 'Bearer ' + tokenKey)
    .then(res=>{
        expect(res.statusCode).to.equal(404);
        done()
    }).catch(err=>done(err))
})

CONSOLE RES: ERROR, Wrong GET request:
TypeError [ERR_INVALID_CHAR]: Invalid character in header content ["Authorization"]

After changing the code to:

it('ERROR, Wrong GET request', (done)=>{
    request(server).get("/api/")
    .auth('Authorization', 'Bearer ' + tokenKey) //**<---here is the change**
    .then(res=>{
        expect(res.statusCode).to.equal(404);
        done()
    }).catch(err=>done(err))
})

CONSOLE RES: OKI ;p

ToniGashi · 2023-12-06T08:49:42Z

In my case: I was trying to get a token from code and when I tried to copy it from the terminal (where I was logging for testing) and pasting it in postman to test if it works, it was adding end-of-line wherever it would have a line end in the terminal so make sure you are not doing that :P

pierreozoux mentioned this issue Mar 10, 2016

error: The header content contains invalid characters matt-major-zz/do-wrapper#24

Closed

simov mentioned this issue Apr 11, 2016

Catch errors from the underlying http module #2164

Merged

simov closed this as completed in #2164 Apr 12, 2016

simov mentioned this issue Apr 12, 2016

Check for self.req existence inside the write method #2165

Merged

hypesystem mentioned this issue Apr 24, 2016

HTTP Client Error ToothlessGear/node-gcm#228

Closed

vladholubiev mentioned this issue Dec 20, 2016

filesDownload: The header content contains invalid characters dropbox/dropbox-sdk-js#89

Closed

pmuens mentioned this issue Feb 13, 2017

Serverless Error: The header content contains invalid characters serverless/serverless#3202

Closed

virtser mentioned this issue Apr 16, 2017

The header content contains invalid characters dorbel-tech/dorbel-app#868

Closed

sindresorhus mentioned this issue Aug 5, 2017

Uncaught error: The header content contains invalid characters sindresorhus/got#353

Closed

lukechilds mentioned this issue Oct 16, 2017

Failures from cache PR sindresorhus/got#392

Closed

2 tasks

dannycochran mentioned this issue Dec 12, 2018

Curl c ommand whichsteveyp/json-proxy-middleware#16

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

request.get crashes with "The header content contains invalid characters" #2120

request.get crashes with "The header content contains invalid characters" #2120

danielgindi commented Mar 8, 2016 •

edited

danielgindi commented Mar 8, 2016

luiselizondo commented Mar 9, 2016

luiselizondo commented Mar 9, 2016

danielgindi commented Mar 9, 2016

pierreozoux commented Mar 10, 2016

TimBeyer commented Mar 24, 2016

TimBeyer commented Mar 24, 2016

jankronquist commented Apr 11, 2016

simov commented Apr 11, 2016

danielgindi commented Apr 11, 2016

TimBeyer commented Apr 11, 2016

simov commented Apr 12, 2016

khurrumqureshi commented Apr 12, 2016

simov commented Apr 12, 2016

khurrumqureshi commented Apr 13, 2016

bitliner commented Nov 8, 2016 •

edited

henhal commented Jan 9, 2017 •

edited

lloyddugmore commented Mar 22, 2018

nkostadinov commented Nov 20, 2018

LeszekChojnacki commented Jan 4, 2020

ToniGashi commented Dec 6, 2023

request.get crashes with "The header content contains invalid characters" #2120

request.get crashes with "The header content contains invalid characters" #2120

Comments

danielgindi commented Mar 8, 2016 • edited

danielgindi commented Mar 8, 2016

luiselizondo commented Mar 9, 2016

luiselizondo commented Mar 9, 2016

danielgindi commented Mar 9, 2016

pierreozoux commented Mar 10, 2016

TimBeyer commented Mar 24, 2016

TimBeyer commented Mar 24, 2016

jankronquist commented Apr 11, 2016

simov commented Apr 11, 2016

danielgindi commented Apr 11, 2016

TimBeyer commented Apr 11, 2016

simov commented Apr 12, 2016

khurrumqureshi commented Apr 12, 2016

simov commented Apr 12, 2016

khurrumqureshi commented Apr 13, 2016

bitliner commented Nov 8, 2016 • edited

henhal commented Jan 9, 2017 • edited

lloyddugmore commented Mar 22, 2018

nkostadinov commented Nov 20, 2018

LeszekChojnacki commented Jan 4, 2020

ToniGashi commented Dec 6, 2023

danielgindi commented Mar 8, 2016 •

edited

bitliner commented Nov 8, 2016 •

edited

henhal commented Jan 9, 2017 •

edited