Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mongoose takes a very long time to save large Maps with subdocuments to the database #13191

Closed
2 tasks done
TheYuriG opened this issue Mar 21, 2023 · 2 comments
Closed
2 tasks done
Milestone

Comments

@TheYuriG
Copy link

TheYuriG commented Mar 21, 2023

Prerequisites

  • I have written a descriptive issue title
  • I have searched existing issues to ensure the bug has not already been reported

Mongoose version

7.0.2

Node.js version

16.9.0

MongoDB server version

latest

Typescript version (if applicable)

No response

Description

Mongoose takes an absurdly long time to save a large Map to the database. This problem doesn't happen if you replace the Maps for Arrays.

This was initially reported at #5925 and requested to be made into a new issue when the reproduction script was created, so here it is.

Steps to Reproduce

Clone my reproduction repo here, then either edit your mongoKey into the index.js file or add a mongoKey.js file to /private so it can be imported by the index.js file.

Once that's done, you can either use npm start to start the script in colored mode or npm run colorless to use colorless mode.

All major steps are color-coded, if you choose to retain the colors, so you can more easily parse the information.

The script will connect to a database and create 20K documents called Minis that only have a small string and a number, then use those to sort and create a Map and an Array version of those documents, sorted by timestamp value.

I've reused the script a few times and I consistently get ~6-minute timers for saving the Maps using mongoose, while saving the Arrays usually takes around ~5-seconds.

Try out the script and let me know if this is an actual bug with the library double checking If every value actually isn't a duplicate of the other value or if this was a PEBCAK.

Expected Behavior

Expected behavior was to not take over 6 minutes to save the Map to the database.

@vkarpov15 vkarpov15 added this to the 7.0.4 milestone Mar 21, 2023
vkarpov15 added a commit that referenced this issue Apr 16, 2023
@vkarpov15 vkarpov15 modified the milestones: 7.0.4, 6.10.6 Apr 16, 2023
vkarpov15 added a commit that referenced this issue Apr 18, 2023
vkarpov15 added a commit that referenced this issue Apr 27, 2023
perf: speed up creating maps of subdocuments
@vkarpov15 vkarpov15 modified the milestones: 6.11.0, 6.11.1, 6.11.2 May 1, 2023
@vkarpov15 vkarpov15 modified the milestones: 6.11.2, 6.11.3 Jun 8, 2023
@vkarpov15 vkarpov15 modified the milestones: 6.11.3, 6.11.4 Jul 12, 2023
vkarpov15 added a commit that referenced this issue Jul 15, 2023
vkarpov15 added a commit that referenced this issue Jul 17, 2023
perf: speed up mapOfSubdocs benchmark by 4x by avoiding unnecessary O(n^2) loop in getPathsToValidate()
@TheYuriG
Copy link
Author

At last, closure! Thanks Valeri!

@vkarpov15
Copy link
Collaborator

Thanks for your patience, took a lot of debugging but the bottleneck should be gone now 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants