Mongoose not reconnecting when MongoDB driver does #6140
Comments
|
I did some digging around in the drivers code and this is what I found: mongodb-core/lib/topologies/mongos.js After a disconnect the self.isConnected() check never evaluates to true, therefore the reconnect event is never fired. |
vkarpov15
added
the
needs repro script
|
@maloguertin does this happen even if you don't use |
|
With this simple script: the results are: reconnected event never fires. I posted an issue to mongodb-core: The driver never fires the reconnect event. Therefore I don't expect this to be a bug on your side, I haven't dug around too much but I'm guessing mongoose relies on the mongodb driver reconnect event to:
I experimented a bit and when I changed the check for |
|
I'm going to mark this is as an underlying library issue for now, please report back on how the driver issue is resolved |
sobafuchs
added
the
underlying library issue
|
Is there any workaround we can use in the meantime? I'm experiencing this a lot lately. Connection state stays at "connecting" (2). Restarting or starting a second instance "solves" the issue immediately. EDIT: function __checkConnection() {
mongoose.connections.forEach((c, i) => {
if (c.readyState === 2 && lastState[i] !== 2) { // hangs in "connecting" state
lastState[i] = c.readyState;
console.warn('forcing manually reconnect of mongoose connection');
Raven.captureMessage('forcing manually reconnect of mongoose connection', {extra: {env: process.env, lastState}});
// reconnect
exports.reconnect()
.catch((err) => {
Raven.captureException('forcing reconnect failed', {extra: {env: process.env, lastState, err}});
clearInterval(watchDog);
process.exit(3);
});
} else if (lastState[i] !== c.readyState) {
lastState[i] = c.readyState;
}
});
}and have an interval running that checks this every second. |
|
@simllll do you have a way to repro this without using docker? It would definitely save me a lot of time if I didn't have to install and figure out docker to try to repro this. |
|
@vkarpov15 I played around and found something "interesting". I don't even need to stop the mongos, it's enough to reduce the socket timeout. This script works without a docker, you need a mongos instance to connect to though. const mongoose = require('mongoose');
const dburl = 'mongodb://127.0.0.1:27017/test';
const testSchema = mongoose.Schema(
{
test: String
},
{
timestamps: true
}
);
const MySchema = mongoose.model('TestSchema', testSchema);
mongoose.connect(dburl, {
// loggerLevel: 'info',
reconnectTries: 2, // increasing doesn't help
reconnectInterval: 1000,
connectTimeoutMS: 2000,
socketTimeoutMS: 2000,
poolSize: 1 // doesn't matter
});
// When successfully connected
mongoose.connection.on('connected', () => {
console.log('dbevent: open');
});
// When successfully reconnected
mongoose.connection.on('reconnected', () => {
console.log('dbevent: reconnected');
});
// If the connection throws an error
mongoose.connection.on('error', (err) => {
console.log(`dbevent: error: ${err}`);
});
// When the connection is disconnected
mongoose.connection.on('disconnected', () => {
console.log('dbevent: disconnected');
});
setInterval(() => {
console.log('READYSTATE', mongoose.connection.readyState);
for (let i = 0; i < 3; i++) {
try {
MySchema.findOne({}).exec()
.then(res => console.log('db result:', res))
.catch(e => console.log(e.message));
} catch (e) {
console.log(e.message);
}
}
}, 10000);this script runs fine with local mongodb, it outputs stuff like this (my collection is empty): but when I point it to a mongos instance, it does this: I never get a result, and readystate is 0, even though if you set loggerLevel to 'info' you see a lot of reconnect events from the underyling driver: Could this be related to the "dbevent: disconnected"? I never call disconnect, I'm not sure why it does a disconnect. let me know if this helps! regards |
|
Somehow I believe this is related to #6117. Today we ran into weird mongoose connection issues, which suddenly stopped. Now I just tried to open a mongo cli and also got connection issues on our test environment. Looked into it, and mongos reports too many open files (the test server has default ulimit settings). So somehow we reached more than 800 connections on our test enviornment. Poolsize is 5 on testing instances. So even if 10 or 20 instances would have been connected, this limit is by far not reached. And finally I see a lot of connection established events in mongos log: doesn't seem normal to me. |
|
Tried to debug it a bit, but I'm kinda stuck. not a single line with "db result: undefined". If i run this query immediately after db connect, I get a result back (once). Any ideas what's going on here? Simon |
|
@simllll are you running with mongos specifically? @vkarpov15 You don't need docker, just a mongos so that the mongodb-core/lib/topologies/mongos.js is used related issue on Jira: NODE-1340 |
|
@maloguertin yes, mongos. No issues with a single mongo instance! |
|
I am experiencing this, too. Downgrading to 4.13.12 and explicitly using the decrepated connection logic by not using Mongo Client fixed it for now ( |
|
This issue still persists with our setup. When using a Mongos instance it can' reconnect. I tried our setup and the above script using mongoose@5.0.18 & mongoose@5.2.5, no reconnect. |
vkarpov15
removed
the
needs repro script
|
+1 |
|
@pelzerim @lixiaolihua running locally I can confirm that the below successfully reconnects. Can you provide more info on what your mongoose connection looks like, like the general form of your connection string and the options you're using? Also, how long is your mongoose process disconnected from the mongos before the mongos comes back up? const mongoose = require('mongoose');
const dburl = 'mongodb://127.0.0.1:51000/test';
const testSchema = mongoose.Schema(
{
test: String
},
{
timestamps: true
}
);
const MySchema = mongoose.model('TestSchema', testSchema);
mongoose.connect(dburl, {
// loggerLevel: 'info',
reconnectTries: 2, // increasing doesn't help
reconnectInterval: 1000,
connectTimeoutMS: 2000,
socketTimeoutMS: 2000,
poolSize: 1 // doesn't matter
});
// When successfully connected
mongoose.connection.on('connected', () => {
console.log('dbevent: open');
});
// When successfully reconnected
mongoose.connection.on('reconnected', () => {
console.log('dbevent: reconnected');
});
// If the connection throws an error
mongoose.connection.on('error', (err) => {
console.log(`dbevent: error: ${err}`);
});
// When the connection is disconnected
mongoose.connection.on('disconnected', () => {
console.log('dbevent: disconnected');
});
setInterval(() => {
console.log('READYSTATE', mongoose.connection.readyState);
for (let i = 0; i < 3; i++) {
try {
MySchema.findOne({}).exec()
.then(res => console.log('db result:', res))
.catch(e => console.log('Error', e.message));
} catch (e) {
console.log('SyncError', e.message);
}
}
}, 10000); |
vkarpov15
added
can't reproduce
|
We had experienced it when Mongo was configured to restart for log rotation. Our mitigation was to use a better method for log rotation that didn't involve restarting Mongo. |
|
@vkarpov15 I adapted the above script to our infrastructure (using ssl): yields |
|
@pelzerim I don't see what the problem is, your output just shows mongoose getting disconnected. Do you kill the mongos that mongoose is connected to and then restart it? |
|
@vkarpov15 Yes, i restarted the container running mongos (takes around 3-5 seconds). I added the actions to the above logs. mongoose or MongoClient never reconnect. Can i provide you with more specific information? |
|
Hmm what does your container setup look like? Can you show your docker-compose file or equivalent? |
|
@vkarpov15 , Why this issue closed? "mongoose": "^5.2.14" Reconnect event never arises. |
|
@yuklia because to the best of our knowledge this issue is fixed. Can you modify the script from #6140 (comment) to repro your issue, and clarify your exact versions of MongoDB and mongoose please? |
|
This seems to have been fixed in the underlying library! The latest mongoose I tested works (5.5.3). |
|
I can confirm that this has been fixed in latest mongo library 3.2.3. Mongoose uses 3.2.2 though? |
|
I am using My connection options looks like this: |
|
@elizatlawy can you please elaborate on how you're determining that no reconnect attempt is made after a disconnection? Do your queries still fail after your MongoDB server is back? |
I have opened a separate ticket for my issue: #14517 |
Do you want to request a feature or report a bug?
bug
What is the current behavior?
When connected to a Mongos container that is restarted Mongoose never reconnects.
I make sure to pass these options:
However the logs show me that the driver does reconnect:
If the current behavior is a bug, please provide the steps to reproduce.
1 Run a nodeJS app that connects to a Mongos on a Docker container
2 Restart your container
What is the expected behavior?
Mongoose should be reconnecting when the container comes back online.
Please mention your node.js, mongoose and MongoDB version.
nodeJS: 8.9.4 and 9.2.0
mongoose: 5.0.5
mongodb: 3.4.10
EDIT:
So I'm using useDb() to connect to other tenant's databases and I just noticed that after the Mongos restart all the calls made to the parent connection db on which I used useDb() works but the calls made on the children connections don't resolve.
Also the readyState is never set to 1 on the parent connection even if the connection is properly reconnected.
The text was updated successfully, but these errors were encountered: