DocumentNotFoundError during save in transaction when using multiple Mongoose modules

[issue-submission]

Do you want to request a feature or report a bug?
Bug

What is the current behavior?
When a document is being saved during a transaction, a DocumentNotFoundError is thrown.

If the current behavior is a bug, please provide the steps to reproduce.
Code:
(simplified)

await connection.transaction(async function fn(session: ClientSession): Promise<void> {
	const entry = await new Entry({ 'identifier': 'entry', 'name': 'name' }).save({ session });
	entry.set('name', 'new name');
	await entry.save({ session }); // DocumentNotFoundError
})

Error:

DocumentNotFoundError: No document found for query "{ _id: new ObjectId("616f3d9e7abcb69d3740739d") }" on model "Entry"
    at C:\Users\User\Documents\Project\node_modules\mongoose\lib\model.js:412:17
    at C:\Users\User\Documents\Project\node_modules\mongoose\lib\model.js:334:9
    at collectionOperationCallback (C:\Users\User\Documents\Project\node_modules\mongoose\lib\drivers\node-mongodb-native\collection.js:168:24)
    at C:\Users\User\Documents\Project\node_modules\mongodb\src\utils.ts:649:5
    at C:\Users\User\Documents\Project\node_modules\mongodb\src\operations\execute_operation.ts:109:9
    at C:\Users\User\Documents\Project\node_modules\mongodb\src\operations\update.ts:166:7
    at C:\Users\User\Documents\Project\node_modules\mongodb\src\cmap\connection_pool.ts:469:13
    at handleOperationResult (C:\Users\User\Documents\Project\node_modules\mongodb\src\sdam\server.ts:600:5)
    at MessageStream.messageHandler (C:\Users\User\Documents\Project\node_modules\mongodb\src\cmap\connection.ts:752:5)
    at MessageStream.emit (node:events:390:28)
    at MessageStream.emit (node:domain:475:12)
    at processIncomingData (C:\Users\User\Documents\Project\node_modules\mongodb\src\cmap\message_stream.ts:167:12)
    at MessageStream._write (C:\Users\User\Documents\Project\node_modules\mongodb\src\cmap\message_stream.ts:64:5)
    at writeOrBuffer (node:internal/streams/writable:389:12)
    at _write (node:internal/streams/writable:330:10)
    at MessageStream.Writable.write (node:internal/streams/writable:334:10) {
  result: {
    acknowledged: true,
    modifiedCount: 0,
    upsertedId: null,
    upsertedCount: 0,
    matchedCount: 0,
    '$where': { _id: new ObjectId("616f3d9e7abcb69d3740739d") }
  },
  numAffected: 0,
  filter: { _id: new ObjectId("616f3d9e7abcb69d3740739d") },
  query: { _id: new ObjectId("616f3d9e7abcb69d3740739d") }
}

The problematic part of the project has not been modified since last time it worked.
Mongoose has been updated. The node_modules folder and the package-lock.json file have been removed before running the npm install command again.

What is the expected behavior?
The entry should be updated accordingly.

What are the versions of Node.js, Mongoose and MongoDB you are using? Note that "latest" is not a version.
Node.js: 16.11.1
Mongoose: 6.0.11
MongoDB: 5.0.2

[issue-submission]

WITH USE OF SHARED LIBRARY
An important point is that in my project, a shared library takes care of the connection to the database. The Connection object is exposed to the other libraries.

I have created a shared library for the purpose of reproduction of the error. Here is the code:

import { connect, connection, Connection, set } from 'mongoose';

set('debug', true);

const DATABASE_CONNECTION_STRING: string = `mongodb://user:password@localhost:27017/db?replicaSet=rs`;

async function Connect(): Promise<Connection> {
    connection.on('open', () => console.log('Connection to database established'));
    connection.on('disconnected', () => console.log('Disconnected from database'));
    connection.on('error', () => {
        console.log('Error occurred during connection to database');
    });
    await connect(DATABASE_CONNECTION_STRING).catch(() => Connect());
    return connection;
}

export { Connect as ConnectToMongoDB };

And here is the code of a clean project that uses the shared library:

import { ClientSession, Connection, Document, Model, Schema, set } from 'mongoose';
import { ConnectToMongoDB } from 'mongoose-project-shared-library';

interface IEntrySchema {
    'identifier': string,
    'name': string
}

const EntrySchema: Schema = new Schema({
    'identifier': {
        type: String,
        required: true,
        unique: true
    },
    'name': {
        type: String,
        required: true
    }
}, { collection: 'entries', minimize: false });
interface IEntryModel extends Document, IEntrySchema { };
let Entry: Model<IEntryModel>;

async function AddEntry(connection: Connection, entry: IEntrySchema): Promise<void> {
    await connection.transaction(async (session: ClientSession) => {
        console.log(`Adding entry: ${entry.identifier}`);
        const entry_document: IEntryModel = await new Entry(entry).save({ session });
        console.log('Entry created');
        console.log(JSON.stringify(entry_document.toJSON()));
        entry_document.set('name', 'New entry name');
        await entry_document.save({ session });
        console.log('Entry modified');
        console.log(JSON.stringify(entry_document.toJSON()));
    });
}

async function Start(): Promise<void> {
    set('debug', true);
    const DATABASE_CONNECTION: Connection = await ConnectToMongoDB();
    Entry = DATABASE_CONNECTION.model<IEntryModel>('Entry', EntrySchema);
    await AddEntry(DATABASE_CONNECTION, { identifier: 'entry', name: 'Entry name' });
}

Start();

The shared library is installed like this:
npm install --save <path_to_shared_library>
In this context, DocumentNotFoundError is thrown.

WITHOUT USE OF SHARED LIBRARY
If I don't use a shared library, no DocumentNotFoundError is thrown. Here is the code I use:

import { ClientSession, connect, connection, Document, model, Model, Schema, set } from 'mongoose';

interface IEntrySchema {
    'identifier': string,
    'name': string
}

const EntrySchema: Schema = new Schema({
    'identifier': {
        type: String,
        required: true,
        unique: true
    },
    'name': {
        type: String,
        required: true
    }
}, { collection: 'entries', minimize: false });
interface IEntryModel extends Document, IEntrySchema { };
const Entry: Model<IEntryModel> = model<IEntryModel>('Entry', EntrySchema);

async function AddEntry(entry: IEntrySchema): Promise<void> {
    await connection.transaction(async (session: ClientSession) => {
        console.log(`Adding entry: ${entry.identifier}`);
        const entry_document: IEntryModel = await new Entry(entry).save({ session });
        console.log('Entry created');
        console.log(JSON.stringify(entry_document.toJSON()));
        entry_document.set('name', 'New entry name');
        await entry_document.save({ session });
        console.log('Entry modified');
        console.log(JSON.stringify(entry_document.toJSON()));
    });
}

async function Connect(): Promise<void> {
    connection.on('open', () => console.log('Connection to database established'));
    connection.on('disconnected', () => console.log('Connection to database ended'));
    connection.on('error', () => {
        console.log('Error during connection');
    });
    await connect('mongodb://user:password@localhost:27017/db?replicaSet=rs').catch(() => Connect());
}

async function Start(): Promise<void> {
    set('debug', true);
    await Connect();
    await AddEntry({ identifier: 'entry', name: 'Entry name' });
}

Start();

Note that if the collection has not been created when the transaction is executed, then the callback is called multiple times, and one of the execution is partial only, as shown below:

Connection to database established
Adding entry: entry
Mongoose: entries.insertOne({ identifier: 'entry', name: 'Entry name', _id: new ObjectId("616fe24325f48a7ffef0701c"), __v: 0}, { session: ClientSession("d28d3a60c8814482bc72e9b3220f139b") })
Mongoose: entries.createIndex({ identifier: 1 }, { unique: true, background: true })
{"identifier":"entry","name":"Entry name","_id":"616fe24325f48a7ffef0701c","__v":0}
Mongoose: entries.updateOne({ _id: new ObjectId("616fe24325f48a7ffef0701c") }, { '$set': { name: 'New entry name' } }, { session: ClientSession("d28d3a60c8814482bc72e9b3220f139b") })
Entry modified
{"identifier":"entry","name":"New entry name","_id":"616fe24325f48a7ffef0701c","__v":0}
Adding entry: entry
Mongoose: entries.insertOne({ identifier: 'entry', name: 'Entry name', _id: new ObjectId("616fe24325f48a7ffef07020"), __v: 0}, { session: ClientSession("d28d3a60c8814482bc72e9b3220f139b") })
Adding entry: entry
Mongoose: entries.insertOne({ identifier: 'entry', name: 'Entry name', _id: new ObjectId("616fe24325f48a7ffef07022"), __v: 0}, { session: ClientSession("d28d3a60c8814482bc72e9b3220f139b") })
Entry created
{"identifier":"entry","name":"Entry name","_id":"616fe24325f48a7ffef07022","__v":0}
Mongoose: entries.updateOne({ _id: new ObjectId("616fe24325f48a7ffef07022") }, { '$set': { name: 'New entry name' } }, { session: ClientSession("d28d3a60c8814482bc72e9b3220f139b") })
Entry modified
{"identifier":"entry","name":"New entry name","_id":"616fe24325f48a7ffef07022","__v":0}

At the end, only one object is created, though.

[issue-submission]

After some tests, I noticed that the problem occurs with version 6.0.8 and following only.

[IslandRhythms]

I'm having a hard time reproducing your problem. When trying to npm install your shared libray, npm looks for a package.json file, not a ts file.

[issue-submission]

Sorry, I only put the TypeScript files here for the sake of clarity, but there are two distinct projects, both having their own package.json file.

Here is the package.json file of the shared library:

{
  "name": "mongoose-project-shared-library",
  "version": "0.0.0",
  "description": "MongooseProjectSharedLibrary",
  "main": "app.js",
  "author": {
    "name": ""
  },
  "scripts": {
    "build": "tsc --build",
    "clean": "tsc --build --clean",
    "start": "tsc && node ./app.js"
  },
  "devDependencies": {
    "@types/node": "^14.14.7",
    "typescript": "^4.0.5"
  },
  "dependencies": {
    "mongoose": "^6.0.8"
  }
}

And here is the package.json file of the library relying on the shared one:

{
  "name": "mongoose-project",
  "version": "0.0.0",
  "description": "MongooseProject",
  "main": "app.js",
  "author": {
    "name": ""
  },
  "scripts": {
    "build": "tsc --build",
    "clean": "tsc --build --clean",
    "start": "tsc && node ./app.js"
  },
  "devDependencies": {
    "@types/node": "^14.14.7",
    "typescript": "^4.0.5"
  },
  "dependencies": {
    "mongoose": "^6.0.8",
    "mongoose-project-shared-library": "file:../MongooseProjectSharedLibrary"
  }
}

So if we have a directory containing two directories: MongooseProjectSharedLibrary (for the shared library) and MongooseProject (for the main project). Then, from the main project, we can link the shared library running the following command:
npm install --save ../MongooseProjectSharedLibrary

[IslandRhythms]

So following your steps, this is what shows up in mongoose-project package.json file
"mongoose-project-shared-library": "file:..\\MongooseProjectSharedLibrary"
and the error message coming from the main file is that it cannot find the module

[issue-submission]

Here is a GitHub repository containing the two libraries: https://github.com/issue-submission/MongooseExample

Install dependencies for MongooseProjectSharedLibrary then for MongooseProject: npm install

In MongooseProject folder, create a file named .env and specify the following variables in order to connect to the database that should be used:

DB_USER=<user name>
DB_PASSWORD=<user password>
DB_HOST=<database host>
DB_PORT=<database port>
DB_NAME=<database name>
DB_CONNECTION_STRING=mongodb://${DB_USER}:${DB_PASSWORD}@${DB_HOST}:${DB_PORT}/${DB_NAME}

Alternatively:

• either edit the app.js file of the MongooseProjectSharedLibrary and specify directly the connection string for DATABASE_CONNECTION_STRING;
• or provide DB_CONNECTION_STRING environment variable before executing the script.

Build MongooseProjectSharedLibrary:
npm run build

Then build and start MongooseProject:
npm run build && npm run start

[vkarpov15]

I did a lot of digging into this and haven't been able to figure out fundamentally why this happens - it looks like the command sent to the MongoDB server has the correct txnNumber set in both cases. However, I did come up with an alternative approach to fix this issue in Mongoose, as well as a couple of workarounds.

1. Make Mongoose a peer dependency (package.json peerDependencies) in MongooseProjectSharedLibrary. This is the approach we recommend. Realistically, we're unable to test cases where objects from one copy of the Mongoose module end up in another copy of the Mongoose module, and while we do our best, there's no guarantee that passing a ClientSession from one copy of Mongoose to another will work as expected.
2. Define EntrySchema in MongooseProjectSharedLibrary. This issue is somehow connected to the fact that you're defining a Schema with one copy of the Mongoose module, and using it to define a model on a different copy of the Mongoose module. We haven't been able to figure out why, but we can confirm that moving EntrySchema to the shared library fixes the issue, even without making mongoose a peer dependency.

f37577d will fix this particular issue for v6.0.14, but we would still recommend making Mongoose a peer dependency in your shared library.