Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"Error: timeout reached: only 0 responses received out of 1" while scaling up a high-traffic system with multiple nodes. #6

Open
OnlySekai opened this issue Jun 8, 2023 · 8 comments

Comments

@OnlySekai
Copy link

OnlySekai commented Jun 8, 2023

I encountered an error "Error: timeout reached: only 0 responses received out of 1" while scaling up a high-traffic system with multiple nodes.

My system run 2 same instances on digitalOceans, and sometime it had been crash with error:

/workspace/node_modules/@socket.io/redis-streams-adapter/dist/cluster-adapter.js:348 reject(new Error(timeout reached: only ${storedRequest.current} responses received out of ${storedRequest.expected})); Error: timeout reached: only 0 responses received out of 1 at listOnTimeout (node:internal/timers:559:17) at processTimers (node:internal/timers:502:7)

my setup redis
`const redisClient = createClient({
url: config.REDIS_ADDRESS,
});

const initRedis = () => {
redisClient.on('error', err => console.log(err));
return redisClient.connect().then(() => console.log('connected redis success'));
};`

my setup adapter is:
Io.io = new Server(httpServer, { adapter: createAdapter(redisClient, { maxLen: 500_000, readCount: 5000, heartbeatInterval: 5_000, heartbeatTimeout: 120_000, }), cors: { origin: config.CORS_WEBSOCKET, }, connectionStateRecovery: { maxDisconnectionDuration: config.MAX_DISCONNECTION_DURATION, skipMiddlewares: true, }, }).use(checkTokenAuth);

my setup use witth express:

const server = http.createServer(app); const io = new Io(); Promise.all([db.init(), initRedis()]).then(async () => { server.listen(config.PORT, () => { logger.info(HttpNetwork is running at port: ${config.PORT}); }); io.startSocket(server, () => console.log(socket is running at port: ${config.PORT})); });

@robertogallardo
Copy link

I had a similar error when making a call to fetchSockets().

Error: timeout reached: only 0 responses received out of 1 at Timeout._onTimeout (node_modules/@socket.io/redis-streams-adapter/dist/cluster-adapter.js:348:28) at listOnTimeout (node:internal/timers:564:17) at process.processTimers (node:internal/timers:507:7) Error: timeout reached: only 0 responses received out of 1 at Timeout._onTimeout (node_modules/@socket.io/redis-streams-adapter/dist/cluster-adapter.js:348:28) at listOnTimeout (node:internal/timers:564:17) at process.processTimers (node:internal/timers:507:7)

The other instances show an error uncaughtException: Maximum call stack size exceeded in the hasBinary function of the util.js file.

My solution was to check for sockets following @darrachequesne's recommendation and not use fetchSockets.
socketio/socket.io#4183 (comment)

@mgarf
Copy link

mgarf commented Aug 7, 2023

I've also been getting a similar error with const recipientsList = await io.in(exSocket.appGuid).fetchSockets();

timeout reached: only 3 responses received out of 7

@hysapp
Copy link

hysapp commented Aug 25, 2023

I have the same problem in version 0.1.0
when call let sockets = await io.fetchSockets() with 4 cpu using node js cluster

a lot of:
timeout reached: only 0 responses received out of 4
.
.
.
timeout reached: only 1 responses received out of 3
/root/daddy/node_modules/@socket.io/redis-streams-adapter/dist/cluster-adapter.js:348
reject(new Error(timeout reached: only ${storedRequest.current} responses received out of ${storedRequest.expected}));
^

Error: timeout reached: only 1 responses received out of 3
at Timeout._onTimeout (/root/daddy/node_modules/@socket.io/redis-streams-adapter/dist/cluster-adapter.js:348:28)
at listOnTimeout (node:internal/timers:573:17)
at process.processTimers (node:internal/timers:514:7)

Node.js v20.5.1
worker 1035162 died

@introspection3
Copy link

@hysapp @OnlySekai hello,sirs,do you resolve this bug?
I met this bug on cluster-adapter too . multiple nodes...

@hysapp
Copy link

hysapp commented Nov 5, 2023

@hysapp @OnlySekai hello,sirs,do you resolve this bug? I met this bug on cluster-adapter too . multiple nodes...

No, I implemented everything myself to be able to save and receive the list of online users, the stream adapter was not optimal at all and had bugs, and I used the sharded adapter.

@introspection3
Copy link

@hysapp @OnlySekai hello,sirs,do you resolve this bug? I met this bug on cluster-adapter too . multiple nodes...

No, I implemented everything myself to be able to save and receive the list of online users, the stream adapter was not optimal at all and had bugs, and I used the sharded adapter.

can you tell me ,how to do it ,dear sir.

darrachequesne added a commit that referenced this issue Feb 21, 2024
The ClusterAdapter class has been moved to [1], so that this adapter
only needs to implement to pub/sub mechanism.

Also, [2] should reduce the number of "timeout reached: only x
responses received out of y" errors, since the fetchSockets() requests
will now succeed even if a server leaves the cluster.

[1]: https://github.com/socketio/socket.io-adapter
[2]: socketio/socket.io-adapter@0e23ff0

Related: #6
@darrachequesne
Copy link
Member

socketio/socket.io-adapter@0e23ff0 (included in version 0.2.0) should reduce the number of those errors.

@hysapp do you remember what kind of bugs you encountered?

@introspection3
Copy link

I think this is not a bug ,but nodejs's event loop is blocked.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants