400 Bad Request closing connection prematurely

[Stroemgren]

I have an issue with Express running behind an Nginx proxy.

It's an API receiving about a 1 million requests a day and some of these requests will always be malformed. An example is a GET parameter with an unencoded space. This will make Express throw a 400 request instantly.

Now, my Nginx proxy will error out with "upstream prematurely closed connection while reading response header from upstream". Upstream, in this case, being the express app.

I have 5 instances of the Express app running, but since Nginx never gets to read this 400 response properly, it will try the same request on the next instance, in turn taking down all 5 instances.

Is there any way to override how Express handles these validation errors? Is it a bug? Or maybe I should configure Nginx differently? In which case I know I'm not in the right place.

[dougwilson]

Hi @Stroemgren sorry you're having problems. To better understand if this is an Express issue or a Node.js issue and to even then be able to provide any kind of recommendation or fix for the issue, we would need instructions of how to replicate the issue so we can take a look. Ideally the following information:

1. Version of Node.js
2. Version of Express
3. An example application to use to see the issue (if node -e 'require('express')().listen(3000)' is not enough to reproduce it)
4. The example request that is causing the issue

Thanks!

[wesleytodd]

Is this the behavior you are seeing?

#3813

I believe this was fixed in a patch version of node 11.4.0 and in 10.14.2.

[Stroemgren]

I'm still on Node 8.10.0 on production servers. Express is 4.17.1.

I'm not sure how to reproduce. It's enough to just start Express and send a request with a GET param containing a space. This should be done with cURL or similar with the url in "", so it won't be encoded.

However, I'm not sure how to test that the connection is not closed prematurely.

One thing I do notice though:
If you setup a simple express app and request it with cURL you won't get any headers for the failign response.

curl -i "http://0.0.0.0:3000?param=foo"

HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: text/html; charset=utf-8
Content-Length: 12
ETag: W/"c-Lve95gjOVATpfV8EL5X4nxwjKHE"
Date: Fri, 13 Sep 2019 00:11:48 GMT
Connection: keep-alive

curl -i "http://0.0.0.0:3000?param=foo bar"

HTTP/1.1 400 Bad Request

I just tested this on Node v10.14.1 and Express 4.17.1

[dougwilson]

Thanks, that helps a lot. In your example, above, that 400 bad Request is happening because Express ever gets the request. It is coming directly from Node.js itself. For example, you can see that exact same 400 Bad Request if you use this app, which doesn't include Express at all:

node -e 'require("http").createServer((req, res) => res.end("hello")).listen(3000)'

You'll have to open an issue with the Node.js project for that issue: https://github.com/nodejs/node

I hope I've gotten you on the right path :)

[Stroemgren]

Thanks so much for the quick answers!

[Stroemgren]

Just in case anyone else get's here.
Upgrading to Node 10.16.3 and making the same curl test as above will then result in:

HTTP/1.1 400 Bad Request
Connection: close

[dougwilson]

That particular change you pointed out (including Connection: close with the 400 Bad Request) was put into the 10.x release line in the 10.16.0 release (https://nodejs.org/en/blog/release/v10.16.0/).

[Stroemgren]

I can also confirm that this fixed the issue with Nginx.

I'm surprised that it's been so tough to find info on this. I can't be the only one who experienced a lot of trouble with it. When the connection was not closed, Nginx would retry on the next instance in the pool and take the failed one out for a set amount of time, basically knocking the whole pool down for, in my case, 10 seconds, every time I received a request with bad params.

Thanks a lot for the quick help!

[dougwilson]

Once you pointed to that change, I feel like I recall seeing talk somewhere about NGINX + Node.js DoS issues somewhere, but I searched all over and couldn't find it.

[Stroemgren]

Yea, DoS would be easy with that setup.