TRANSPORT_HANDSHAKE_ERROR after upgrading to socket.io v4

[syntheticencounters]

Describe the bug
After upgrading to socket.io v4 from v2, our client and server connections no longer work correctly. We upgraded the socket.io-client package for the client to the latest version (4.7.3) and upgraded the socket.io server package to the latest version (4.7.3). After doing so, we receive TRANSPORT_HANDSHAKE_ERROR whenever trying to connect using a websocket transport. We are able to connect using polling but the websocket transport fails constantly. The connection appears to work locally without any issues but does not work when pushed to production. Our production infrastructure for the socket io logic consists of a single EC2 instance behind an Application Load Balancer. We route the requests on port 80 through an Apache reverse proxy to an instance within our VPC on port 4500. Once the request hits the socket io instance on port 4500, express and Node JS take over.

To Reproduce
Install latest version of socket.io-client and socket.io, run on an EC2 instance behind an application load balancer, attempt to connect using websocket as the primary transport. We've reproduced the issue using a socket.io-client in a React Native app and through the browser using Safari 16.6. The socket.io server runs Amazon Linux AMI 2 and we use Node JS and express to serve up the socket io application.

Expected behavior
The client and server should complete a handshake.

Platform:
-Safari 16.6
-iOS Simulator 17.0
-iPhone 15 Pro iOS 17.3
-Amazon Linux AMI 2

Additional context
Sharing code will be difficult due to the purpose of the code but I'm happy to do what I can to provide as much information as possible.

[syntheticencounters]

Are there any general recommendations or troubleshooting steps not found on the connection troubleshooting page that we could try in the mean time? Our production services have been down for a few days and I'd like to be as proactive as possible while this issue is being investigated. Thank you.

[darrachequesne]

Hi!

The TRANSPORT_HANDSHAKE_ERROR error can be found in two places in the source code:

• https://github.com/socketio/engine.io/blob/ff0fbfb61f2509ef302870cfd993344c1d035e7d/lib/server.ts#L469-L477
• https://github.com/socketio/engine.io/blob/ff0fbfb61f2509ef302870cfd993344c1d035e7d/lib/server.ts#L312-L317

The first case seems unlikely. The second case means the server receives an invalid WebSocket upgrade request, which could indicate that something between the client and the server may not configured properly and does not forward the WebSocket headers.

Could you please check whether Apache is properly configured?

Reference: https://socket.io/docs/v4/reverse-proxy/#apache-httpd

That would be weird though, since you suggest that it worked previously in v2.

Since you are listening to the connection_error event on the server, you should have access to the req object. Could you please provide the request headers?

io.engine.on("connection_error", (err) => {
  console.log(err.req.headers);
});

[syntheticencounters]

Thanks for the response.

We had a reverse proxy in place prior to the upgrade but I changed the proxy settings during troubleshooting to match what is listed on the socket.io reverse proxy docs for apache. I agree that the second case seems more likely given that the issue isn't present during local testing.

Attached are the headers received in the connection error callback.

host: '172.31.10.182:4500', 'x-forwarded-for': '68.61.25.135, 172.31.19.165', 'x-forwarded-proto': 'https', 'x-forwarded-port': '443', 'x-amzn-trace-id': 'Root=1-659c250c-5c0999ef4b69e3df4ea3eb0c', cookie: 'AWSALBTG=ubJfzmB7aUT9o3j/3Y8GE9LAfvvWVavyzn96jNxOSdNulEBj8bjL6CIxONrOHOr7uPRu1m3jbr/cYI4euyUnb0xyTIJzOh45ATj9+5njLPImJ8Sjfwr2pFrynC7uXumOLxF/kNPXJGTUF6W8mnB3YRkWRS7dY0tTNGyCagVbCuyp; AWSALBTGCORS=ubJfzmB7aUT9o3j/3Y8GE9LAfvvWVavyzn96jNxOSdNulEBj8bjL6CIxONrOHOr7uPRu1m3jbr/cYI4euyUnb0xyTIJzOh45ATj9+5njLPImJ8Sjfwr2pFrynC7uXumOLxF/kNPXJGTUF6W8mnB3YRkWRS7dY0tTNGyCagVbCuyp', 'sec-websocket-key': '+dEkC4tBbdtxfDmhSUElQA==', 'sec-websocket-version': '13', origin: 'https://api.aftplatform.com', 'x-forwarded-host': 'api.aftplatform.com', 'x-forwarded-server': 'api.aftplatform.com', connection: 'close'

[darrachequesne]

Thanks for the quick response! It seems the Upgrade: websocket and Connection: Upgrade headers are missing, so the server understands the request as a classic HTTP request and not as a WebSocket upgrade.

Could you please try to run the Socket.IO server on port 80 (without reverse proxy)?

We have an example with Apache here, though it's within a Socket.IO cluster.

[syntheticencounters]

Running the socket.io express application on port 80 seems to solve the issue. Do you have any ideas why the reverse proxy would be causing problems? Currently we are using what is listed in the socket.io docs.

[syntheticencounters]

I refactored the server logic that routes requests to our reverse proxy and was able to get the handshake to start working again. I've noticed though that we're getting a lot of disconnections with the reason "forced close". It's possible these were there prior to upgrading but we did not previously have a disconnection callback registered for debugging. The force close is caused by the following line. I've removed the namespace from the output out of respect for our source code's client.

socket.io:client invalid state (packet type: { type: 0, nsp: '/namespace_redacted' })