Is the package not using cache properly in our ci runs ?

[AnthonyLenglet]

Hi !

I've been noticing that our ci has been really slow due to a mongodb download happening despite it being seemingly found in the cache folder

Mongodb-Memory-Server* checking MongoDB binaries
Downloading MongoDB "4.4.9": 0% (0mb / 68.7mb)
Downloading MongoDB "4.4.9": 100% (68.7mb / 68.7mb)
Mongodb-Memory-Server* found binary: "/home/runner/.cache/mongodb-binaries/mongod-x64-ubuntu-4.4.9"

PASS src/projects/tests/projects.service.spec.ts (36.746 s)
  ● Console

    console.log
      Downloading MongoDB 4.4.9: 0 % (0mb / 68.6mb)

      at MongoBinaryDownload.Object.<anonymous>.MongoBinaryDownload.printDownloadProgress (node_modules/mongodb-memory-server-core/src/util/MongoBinaryDownload.ts:424:15)

    console.log
      Downloading MongoDB 4.4.9: 40.7 % (27.9mb / 68.6mb)

      at MongoBinaryDownload.Object.<anonymous>.MongoBinaryDownload.printDownloadProgress (node_modules/mongodb-memory-server-core/src/util/MongoBinaryDownload.ts:424:15)

    console.log
      Downloading MongoDB 4.4.9: 82.3 % ([56](https://github.com/Passiverecords/Oney-API/runs/7977031901?check_suite_focus=true#step:5:57).5mb / [68](https://github.com/Passiverecords/Oney-API/runs/7977031901?check_suite_focus=true#step:5:69).6mb)

      at MongoBinaryDownload.Object.<anonymous>.MongoBinaryDownload.printDownloadProgress (node_modules/mongodb-memory-server-core/src/util/MongoBinaryDownload.ts:424:15)

our jest-mongodb-config.ts file currently looks like this

export default {
  mongodbMemoryServerOptions: {
    binary: {
      version: '4.4.9',
      skipMD5: true,
    },
    autoStart: false,
    instance: {},
  },
};

is somethign wrong with the way it is currently set up ?

[hasezoey]

weird that it does not use the downloaded binary but needs to re-download it in the tests
could you provide:

• what ci are you using (github actions / gitlab runner / travis, etc)
• where are you applying mongodbMemoryServerOptions to? (because this is not read normally by MMS itself)

I've been noticing that our ci has been really slow due to a mongodb download happening despite it being seemingly found in the cache folder

from what i know, most ci runners dont cache ~/.cache by default, and would need to be done by the user (though in MMS's case i would recommend to explicitly set the DOWNLOAD_DIR config option and cache that)

[AnthonyLenglet]

We're using github actions to run the ci with the following steps:

steps:
      - name: Load repository
        uses: actions/checkout@master
        with:
          persist-credentials: false
      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: 12
          cache: 'npm'
        env:
          REGISTRY_TOKEN: ${{ secrets.REGISTRY_TOKEN }}
      - name: Run tests
        timeout-minutes: 30
        env:
          REGISTRY_TOKEN: ${{ secrets.REGISTRY_TOKEN }}
        run: |
          npm ci
          npm run test:cov

As for mongodbMemoryServerOptions, this is coming from @shelf/jest-mongodb, a jest preset for this package

from what i know, most ci runners dont cache ~/.cache by default, and would need to be done by the user (though in MMS's case i would recommend to explicitly set the DOWNLOAD_DIR config option and cache that)

I wonder if something about this changed, since re-running the job for the CI still tells me that the binary was found in ~/.cache, despite me not having set DOWNLOAD_DIR, maybe github action is one of the few that caches it ?

[hasezoey]

since re-running the job for the CI still tells me that the binary was found in ~/.cache, despite me not having set DOWNLOAD_DIR

if the job is not based on the output of another job, then this is weird, because from what i know github actions does not cache anything by default, though environments may be re-used on re-running (though i am unsure about that)

despite me not having set DOWNLOAD_DIR

i just mentioned to set DOWNLOAD_DIR when caching to have a explicit folder set to cache, even if the default may change in MMS

[AnthonyLenglet]

Quick update, I found this issue when attempting to update our current CI setup, and I just found out that the issue only happens when using npm ci instead of npm install

with npm install:

> mongodb-memory-server@7.6.3 postinstall /home/runner/work/<PROJECT>/node_modules/@shelf/jest-mongodb/node_modules/mongodb-memory-server
> node ./postinstall.js

Mongodb-Memory-Server* checking MongoDB binaries
Downloading MongoDB "4.4.9": 0% (0mb / 68.7mb)
Downloading MongoDB "4.4.9": 100% (68.7mb / 68.7mb)
Mongodb-Memory-Server* found binary: "/home/runner/.cache/mongodb-binaries/mongod-x64-ubuntu-4.4.9"

> @nestjs/core@7.6.18 postinstall /home/runner/work/<PROJECT>/node_modules/@nestjs/core
> opencollective || exit 0


> husky@4.3.8 postinstall /home/runner/work/<PROJECT>/node_modules/husky
> opencollective-postinstall || exit 0


> mongodb-memory-server@6.9.3 postinstall /home/runner/work/<PROJECT>/node_modules/mongodb-memory-server
> node ./postinstall.js

mongodb-memory-server: checking MongoDB binaries cache...
Downloading MongoDB 4.4.9: 0 % (0mb / 68.6mb)
mongodb-memory-server: binary path is /home/runner/.cache/mongodb-binaries/4.4.9/mongod

...

PASS src/projects/tests/projects.service.spec.ts (22.532 s)

with npm ci:

> mongodb-memory-server@6.9.3 postinstall /home/runner/work/<PROJECT>/node_modules/mongodb-memory-server
> node ./postinstall.js

mongodb-memory-server: checking MongoDB binaries cache...
Downloading MongoDB 4.4.9: 0 % (0mb / 68.6mb)
mongodb-memory-server: binary path is /home/runner/work/<PROJECT>/node_modules/.cache/mongodb-memory-server/mongodb-binaries/4.4.9/mongod

> husky@4.3.8 install /home/runner/work/<PROJECT>/node_modules/husky
> node husky install

husky > Setting up git hooks
CI detected, skipping Git hooks installation.
husky > Done

> husky@4.3.8 postinstall /home/runner/work/<PROJECT>/node_modules/husky
> opencollective-postinstall || exit 0


> mongodb-memory-server@7.6.3 postinstall /home/runner/work/<PROJECT>/node_modules/@shelf/jest-mongodb/node_modules/mongodb-memory-server
> node ./postinstall.js

Mongodb-Memory-Server* checking MongoDB binaries
Downloading MongoDB "4.4.9": 0% (0mb / 68.7mb)
Downloading MongoDB "4.4.9": 100% (68.7mb / 68.7mb)
Mongodb-Memory-Server* found binary: "/home/runner/.cache/mongodb-binaries/mongod-x64-ubuntu-4.4.9"

...

PASS src/projects/tests/projects.service.spec.ts (35.925 s)
  ● Console

    console.log
      Downloading MongoDB 4.4.9: 0 % (0mb / 68.6mb)

      at MongoBinaryDownload.Object.<anonymous>.MongoBinaryDownload.printDownloadProgress (node_modules/mongodb-memory-server-core/src/util/MongoBinaryDownload.ts:424:15)

    console.log
      Downloading MongoDB 4.4.9: 41.5 % (28.5mb / 68.6mb)

      at MongoBinaryDownload.Object.<anonymous>.MongoBinaryDownload.printDownloadProgress (node_modules/mongodb-memory-server-core/src/util/MongoBinaryDownload.ts:424:15)

    console.log
      Downloading MongoDB 4.4.9: 84.4 % (57.9mb / 68.6mb)

      at MongoBinaryDownload.Object.<anonymous>.MongoBinaryDownload.printDownloadProgress (node_modules/mongodb-memory-server-core/src/util/MongoBinaryDownload.ts:424:15)

the cache path is /home/runner/work/<PROJECT>/node_modules/.cache in npm ci and /home/runner/.cache in npm install, yet it seems to give out Mongodb-Memory-Server* found binary: "/home/runner/.cache/mongodb-binaries/mongod-x64-ubuntu-4.4.9" in both cases ? I wonder if its a cache coming from the previous job runs ?

[hasezoey]

mongodb-memory-server@6.9.3 postinstall /home/runner/work//node_modules/mongodb-memory-server
mongodb-memory-server@7.6.3 postinstall /home/runner/work//node_modules/@shelf/jest-mongodb/node_modules/mongodb-memory-server

you seem to have 2 different versions installed and set-up, try running npm ls mongodb-memory-server or yarn why mongodb-memory-server and try to resolve only using one version

also 8.9.0 is currently the latest version and those old versions had different resolving and binary naming schemes

Note: i dont know what version of @shelf/jest-mongodb your use, but @shelf/jest-mongodb@4.1.0 supports MMS 8.8.0 from what i can tell

[AnthonyLenglet]

I spent some time updating all the packages in our project and was able to update @shelf/jest-mongodb and remove the duplicated mongodb-memory-server, and the issue is no longer happening !

I originally kept both mongodb-memory-server and @shelf/jest-mongodb in order to move the tests from one to the other one by one over time, but I just ended up moving everything else over now