Strapi unit tests failing

[kasonde]

The Strapi Unit Testing docs don't produce the expected results
I've attempted to write unit tests in strapi following the unit testing docs for the latest strapi version and to no avail. Unit tests fail when you attempt to test using an env that isn't the default environment strapi runs in.

Steps to reproduce the behavior

1. Go to strapi unit testing docs and create a new project following the guide immediately after
2. Run 'yarn test'
3. See error below

yarn run v1.22.4
$ jest --forceExit --detectOpenHandles
  ●  process.exit called with "1"

       7 |   if (!instance) {
       8 |     /** the following code in copied from `./node_modules/strapi/lib/Strapi.js` */
    >  9 |     await Strapi().load();
         |     ^
      10 |     instance = strapi; // strapi is global now
      11 |     await instance.app
      12 |       .use(instance.router.routes()) // populate KOA routes

      at Strapi.stop (node_modules/strapi/lib/Strapi.js:263:13)
      at node_modules/strapi/lib/Strapi.js:391:16
          at async Promise.all (index 3)
      at Strapi.runBootstrapFunctions (node_modules/strapi/lib/Strapi.js:394:5)
      at Strapi.load (node_modules/strapi/lib/Strapi.js:326:5)
      at setupStrapi (tests/helpers/strapi.js:9:5)
      at Object.<anonymous> (tests/app.test.js:8:3)

 RUNS  tests/app.test.js
error Command failed with exit code 1.

Expected behavior
Tests must pass

System

• Node.js version: v12.16.3
• NPM version: 6.14.5
• Strapi version: 3.1.4
• Database: sqlite
• Operating system: MacOS Catalina 10.15.6

[dtmzr]

Got the same output. Each of these changes make the test pass:

• Remove config/env/test/database.json
• Change the await Strapi().load() call to await Strapi()

So I would assume something with the database configuration is not working as expected.

Any thoughts from some more experienced users about this?

[kasonde]

After some checking it seems strapi hates it if you have database.json in your particular config/env/ folder.

I found for me, changing it to database.js matching the same format as the default database.js in the default config makes the tests run.

The other issue now is the sqlite db that's generated (using sqlite for my test db) doesn't get deleted or obey my specified file name. It generates data.db as opposed to what I specified

[qunabu]

@kasonde please try to remove all the options from ./config/env/test/database.json so it would only contain

{
  "defaultConnection": "default",
  "connections": {
    "default": {
      "connector": "bookshelf",
      "settings": {
        "client": "sqlite",
        "filename": ".tmp/test.db"
      }
    }
  }
}

@dtmzr

Remove config/env/test/database.json

This would make test run on you current database which can lead to serious problems.

[glascar]

Same issue solved removing options as @qunabu said!

[marefati110]

id do all thing that @kasonde said.
but not worked

databse: mysql

@derrickmehaffy its look like a hotfix

[qunabu]

@marefati110 what strapi version you have ?

[marefati110]

@qunabu
strapi: 3.2.0
databse: mysql
os: centos7

[kcouliba]

Encounter same issue here. Any update ?
I manage to make it work though connecting to my postgresql database.

[qunabu]

I've managed to create and example for latest strapi 3.3.2 and postgres database. It's attached to CircleCI so it should work with most of the CI as well

[saulmmendoza]

The repo from @qunabu works great, I think those files should be on Strapi by default, it was a pain in the ass to debug/start testing

[Barbapapazes]

@saulmmendoza I agree, I'm trying strapi but these many errors are so disappointing ☹

[Barbapapazes]

and it still doesn't work! please, explain how you do to test your app!

[qunabu]

I've noticed one thing in my CI/CD environment.

The same tests are failing for one of the gitlab runner and pass with other. The difference between those is they are the same runners based on ubuntu image but launched on other machines.

One that pass has more resources (cores and ram) then one that occasionally fails.

Some of the packages seems to be not waiting for a callback when you have less resources, or CI/CD needs really great machine to run them.

Anyway this can be address better in terms of feedback from the framework.

[blind675]

Managed to make it work with sqlite and config

defaultConnection: 'default',
 connections: {
  default: {
   connector: 'bookshelf',
     settings: {
       client: 'sqlite',
       filename: env('DATABASE_FILENAME', '.tmp/data.db'),
     },
   options: {
     useNullAsDefault: true,
      pool: {
       min: 0,
       max: 1
      }
    },
   },
 },

by installing knex and sqlight3 npm modules. It seems more of a patch than a fix.

[lucap84]

In my experience I followed "Unit testing" Strapi docs and I received the mentioned error at the same row:

process.exit called with "1" during await Strapi().load();

but looking at error stack I see a clear error during the database initialization:

> NODE_ENV=test jest --forceExit --detectOpenHandles

  ●  process.exit called with "1"

      at Strapi.stop (node_modules/strapi/lib/Strapi.js:303:13)
      at node_modules/strapi-connector-bookshelf/lib/knex.js:189:16
      at node_modules/strapi-connector-bookshelf/node_modules/lodash/lodash.js:4925:15
      at baseForOwn (node_modules/strapi-connector-bookshelf/node_modules/lodash/lodash.js:2990:24)
      at node_modules/strapi-connector-bookshelf/node_modules/lodash/lodash.js:4894:18
      at Function.forEach (node_modules/strapi-connector-bookshelf/node_modules/lodash/lodash.js:9368:14)
      at Object.<anonymous>.module.exports (node_modules/strapi-connector-bookshelf/lib/knex.js:37:5)
      at Object.initialize (node_modules/strapi-connector-bookshelf/lib/index.js:44:5)
      at Object.initialize (node_modules/strapi-database/lib/connector-registry.js:30:25)
      at DatabaseManager.initialize (node_modules/strapi-database/lib/database-manager.js:36:27)

🥇 I resolved this installing some missing dependencies (not mentioned on docs): strapi-connector-bookshelf and knex

here the other configurations:
package.json:

"devDependencies": {
    "jest": "^26.6.3",
    "jest-each": "^26.6.2",
    "knex": "^0.95.4",
    "sqlite3": "^5.0.2",
    "strapi-connector-bookshelf": "3.5.3",
    "supertest": "^6.1.3"
  },
"jest": {
    "testPathIgnorePatterns": [
      "/node_modules/",
      ".tmp",
      ".cache"
    ],
    "testEnvironment": "node"
  }

config/env/test/database.js:

module.exports = ({ env }) => ({
  defaultConnection: "default",
  connections: {
    default: {
      connector: "bookshelf",
      settings: {
        client: "sqlite",
        filename: ".tmp/test.db"
      },
      options: {
        useNullAsDefault: true,
          pool: {
            min: 0,
            max: 1
        }
      },
    }
  }
});

my Strapi version is 3.5.3

[innerdvations]

Running Strapi 3.5.4, I think I figured it out.

I have the exact same error when using Postgres/bookshelf. Strapi runs completely fine normally with postgres. There aren't any missing modules. Using the exact same database config for tests fails on Strapi().load(), and removing .load() doesn't work for me.

If I change the database connector to Mongo/mongoose, tests run fine with no error.

So I threw it all in Docker and watched the output while it tried running the tests, and right before it fails I got a bunch of max connections errors from the Postgres server. I upped the max connections to 200 and it worked. So it seems that Strapi().load() opens too many connections to Postgres and then crashes when it's refused.

[raulfdm]

I'm having the exact same problem:

process.exit called with "1"

      10 |     instance = strapi; // strapi is now global
      11 |     await instance.app
    > 12 |       .use(instance.router.routes()) // populate KOA routes
         |     ^
      13 |       .use(instance.router.allowedMethods()); // populate KOA methods
      14 |     instance.server = http.createServer(instance.app.callback());
      15 |   }

      at Strapi.stop (node_modules/strapi/lib/Strapi.js:316:13)
      at node_modules/strapi/lib/Strapi.js:450:18
          at async Promise.all (index 5)
      at Strapi.runLifecyclesFunctions (node_modules/strapi/lib/Strapi.js:443:5)
      at Strapi.load (node_modules/strapi/lib/Strapi.js:379:5)
      at setupStrapi (tests/helpers/strapi.js:12:5)
      at tests/app.test.js:12:3
[2021-10-07T12:21:21.936Z] error bootstrap function in plugin "users-permissions" failed
[2021-10-07T12:21:21.936Z] error undefined

Tried all proposed solutions but none seems to work. 🤔

Update

I've figured out to work. I basically:

1. Added a package called strapi-provider-upload-local to override my production upload provider
2. inside /config/env/test, have created a file plugins.js to override my plugins and defined:

     module.exports = ({ env }) => ({
       upload: {
         provider: 'local',
         "providerOptions": {
           "sizeLimit": 100000
         }
       },
     });

3. In my test/app.test.js, before the test coded increase jestTimeout => jest.setTimeout(30000)

And now it seems working.

Although the error messages are not clear at all. I wish somehow we could have clearer error messages but maybe it's a combination of Strapi + Jest

[iSeiryu]

@raulfdm could you share your package.json?
There are probably lots of versions of this same error. They just don't bubble up properly so all errors look more or less the same.

#12 47.83 [2021-12-15T21:10:32.779Z] error bootstrap function in plugin "users-permissions" failed
#12 47.84 [2021-12-15T21:10:32.790Z] error KnexTimeoutError: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?
#12 47.84     at Client_SQLite3.acquireConnection (/srv/node_modules/knex/lib/client.js:348:26)
#12 47.84     at runNextTicks (internal/process/task_queues.js:58:5)
#12 47.84     at listOnTimeout (internal/timers.js:523:9)
#12 47.84     at processTimers (internal/timers.js:497:7)
#12 47.84     at Transaction.acquireConnection (/srv/node_modules/knex/lib/transaction.js:212:28)
#12 47.84     at Object.create (/srv/node_modules/strapi-connector-bookshelf/lib/queries.js:58:14)
#12 47.84     at Object.create (/srv/node_modules/strapi-database/lib/queries/helpers.js:15:18)
#12 47.84     at async Promise.all (index 178)
#12 47.84     at Object.updatePermissions (/srv/node_modules/strapi-plugin-users-permissions/services/UsersPermissions.js:317:7)
#12 47.84     at Object.<anonymous>.module.exports (/srv/node_modules/strapi-plugin-users-permissions/config/functions/bootstrap.js:214:3)
#12 47.84     at async Promise.all (index 6)
#12 47.84     at Strapi.runLifecyclesFunctions (/srv/node_modules/strapi/lib/Strapi.js:443:5)
#12 47.84     at Strapi.load (/srv/node_modules/strapi/lib/Strapi.js:379:5)
#12 47.84     at setupStrapi (/srv/test/helpers/strapi.js:9:5)
#12 47.84     at /srv/test/cron.test.js:9:5
#12 47.85   ●  process.exit called with "1"
#12 47.85
#12 47.85        7 |   if (!instance) {
#12 47.85        8 |     /** the following code in copied from `./node_modules/strapi/lib/Strapi.js` */
#12 47.85     >  9 |     await Strapi().load();
#12 47.85          |     ^
#12 47.85       10 |     instance = strapi; // strapi is global now
#12 47.85       11 |     await instance.app
#12 47.85       12 |       .use(instance.router.routes()) // populate KOA routes
#12 47.85
#12 47.85       at Strapi.stop (node_modules/strapi/lib/Strapi.js:316:13)
#12 47.85       at node_modules/strapi/lib/Strapi.js:450:18
#12 47.85           at async Promise.all (index 6)
#12 47.85       at Strapi.runLifecyclesFunctions (node_modules/strapi/lib/Strapi.js:443:5)
#12 47.85       at Strapi.load (node_modules/strapi/lib/Strapi.js:379:5)
#12 47.85       at setupStrapi (test/helpers/strapi.js:9:5)
#12 47.85       at test/cron.test.js:9:5

My stripped out package.json (I do not bootstrap appinsights when running tests).

{
  "scripts": {
    "develop": "strapi develop --watch-admin",
    "start": "strapi start",
    "build": "strapi build",
    "strapi": "strapi",
    "test": "jest --forceExit --detectOpenHandles --verbose"
  },
  "dependencies": {
    "applicationinsights": "^2.1.9",
    "applicationinsights-native-metrics": "^0.0.6",
    "knex": "^0.21.18",
    "pg": "8.7.1",
    "strapi": "3.6.8",
    "strapi-admin": "3.6.8",
    "strapi-connector-bookshelf": "3.6.8",
    "strapi-plugin-content-manager": "3.6.8",
    "strapi-plugin-content-type-builder": "3.6.8",
    "strapi-plugin-email": "3.6.8",
    "strapi-plugin-i18n": "3.6.8",
    "strapi-plugin-sync-roles-permissions": "^0.4.3",
    "strapi-plugin-upload": "3.6.8",
    "strapi-plugin-users-permissions": "3.6.8",
    "@strapi/provider-upload-local": "^4.0.0",
    "strapi-provider-upload-azure-storage": "^1.1.3",
    "strapi-utils": "3.6.8"
  },
  "engines": {
    "node": ">=10.16.0 <=14.x.x",
    "npm": ">=6.0.0"
  },
  "jest": {
    "testPathIgnorePatterns": [
      "/node_modules/",
      ".tmp",
      ".cache"
    ],
    "testEnvironment": "node"
  },
  "devDependencies": {
    "jest": "^27.4.5",
    "sqlite3": "^5.0.2",
    "supertest": "^6.1.6"
  }
}

I do have env overrides defined here and they are the same as suggested above (I tried multiple variations).

[iSeiryu]

So, I get the error above when I run tests during the docker build step via

RUN npm run test

But if I comment out await setupStrapi(); and let the image build and then create a container off of it and then run npm run test it suddenly works.

Also it seems to work sometimes (1 out of 100 times) and other tests that do not require strapi setup work just fine.

UPDATE: running setupStrapi() in a container fails too from time to time. Sounds like there is an issue between pg, knex, and node packages. I cannot figure out the right versions that like each other...

UPDATE: running it like this works...

RUN npm run test || true
RUN npm run test

It crashes on the first run but works on the second run. It could be that knex/pg need more time to initialize but I cannot find a way to extend the timeout. Another person stated here knex/knex#3912 that the 1st connection fails.

[emahuni]

The way that Strapi is being used to create test harness is just not good. I am writing a Strapi 4 plugin that I hope strapi team will adopt that simplify testing. It uses Vitest instead of Jest. It's super fast, write tests with Typescript or ESM etc, easy to work with as it is a superset of Jest... It enables us to run any test file without requiring files in app.test.js, and use just one strapi instance, which is loaded on test harness setup. You can also use it to watch for changes. It's a simplified way of testing with Strapi.

I think most issues starts from the fact that we are doing strapi().load() instead of just doing what yarn strapi start or yarn strapi develop is doing. I have done that and everything works just fine, I am just doing this:

import StrapiStart from '@strapi/strapi/lib/commands/start';
await StrapiStart({});

Then start the testing as: NODE_ENV=test yarn test

This just starts strapi as usual, but within a test environment. I think this should work for setting up a server process as well with a production env set of course.
import StrapiStart from '@strapi/strapi/lib/commands/develop'; has too many shenanigans happening with workers, may or may not use it, but start is working just fine.

The SQLite issue

As for the main sqlite question here, I have just also noticed this issue too. So documentation should say:

path.resolve(process.env.PWD, env('DATABASE_FILENAME', '.tmp/data.db'))

instead of

path.join(__dirname, '..', '..', env('DATABASE_FILENAME', '.tmp/data.db'))

This works fine if the database file is coming from /config/env/database.js but not when coming from the dist dir or somewhere else, especially when dealing with TS. So we just need to refer this from the root not from where the file is.

[emahuni]

I'm having the exact same problem:

process.exit called with "1"

      10 |     instance = strapi; // strapi is now global
      11 |     await instance.app
    > 12 |       .use(instance.router.routes()) // populate KOA routes
         |     ^
      13 |       .use(instance.router.allowedMethods()); // populate KOA methods
      14 |     instance.server = http.createServer(instance.app.callback());
      15 |   }

      at Strapi.stop (node_modules/strapi/lib/Strapi.js:316:13)
      at node_modules/strapi/lib/Strapi.js:450:18
          at async Promise.all (index 5)
      at Strapi.runLifecyclesFunctions (node_modules/strapi/lib/Strapi.js:443:5)
      at Strapi.load (node_modules/strapi/lib/Strapi.js:379:5)
      at setupStrapi (tests/helpers/strapi.js:12:5)
      at tests/app.test.js:12:3
[2021-10-07T12:21:21.936Z] error bootstrap function in plugin "users-permissions" failed
[2021-10-07T12:21:21.936Z] error undefined

Tried all proposed solutions but none seems to work. 🤔
...
Although the ### error messages are not clear at all. I wish somehow we could have clearer error messages but maybe it's a combination of Strapi + Jest

To see what is happening do this:

NODE_ENV=test yarn strapi dev

It will spit out what testing suite is swallowing.

[emahuni]

I have published the package as strapi-plugin-vitest on npm, try it out, watch not working as expected, but can use a workaround with chokidar. add this to scripts section of package.json:
"test:w": "chokidar \"**/dist/**\" \"**/tests/**\" -c \"yarn test\" --initial"
then invoke by:
yarn test:w
it's a quick flaky hack, but it will get the job done, just don't update too many times at once.

[qunabu]

I see that there are some test and it's documentation issues because there are breaking changes between version, and to be honest its impossible to maintain the documentation. There is no no focus on backward compatibility - what used to work in 4.2 is not working 4.3 and so on.

@emahuni What you did is just great, kudos

I'm maintaining this repository for both v3 and v4 of Strapi which is good starting point for some people that implement test in their strapi applications. https://github.com/qunabu/strapi-unit-test-example

Can you guide me how to implement your code in this example, is it worth is. Maybe you would be able to create a Pull Request ?

[emahuni]

I see that there are some test and it's documentation issues because there are breaking changes between version, and to be honest its impossible to maintain the documentation. There is no no focus on backward compatibility - what used to work in 4.2 is not working 4.3 and so on.

@emahuni What you did is just great, kudos

I'm maintaining this repository for both v3 and v4 of Strapi which is good starting point for some people that implement test in their strapi applications. https://github.com/qunabu/strapi-unit-test-example

Can you guide me how to implement your code in this example, is it worth is. Maybe you would be able to create a Pull Request ?

Yes that'd be great. Will pull soon.

[QuentinLeCaignec]

I'm honestly a bit confused as to how a simple working setup for Jest + Typescript isn't provided by default or not covered in the docs for v4. I've been searching for a while and still can't get things to work properly, and it all seems like way too much effort just to have some tests for my app (which should be a default).

I'm very interested in what you're doing @emahuni ! Would love to be able to setup vitest, feels like it would be easier to setup and run if it was properly integrated.

[Maxservais]

I am facing the same issue as you @QuentinLeCaignec. Using Typsecript with strapi v4 and been struggling for hours to set up simple unit tests with Jest. Following official docs leads nowhere.. Did you find a way to make it work?

[QuentinLeCaignec]

I am facing the same issue as you @QuentinLeCaignec. Using Typsecript with strapi v4 and been struggling for hours to set up simple unit tests with Jest. Following official docs leads nowhere.. Did you find a way to make it work?

Not really sadly. I have a working jest + TS with ts-jest, but initialising the Strapi instance or testing an endpoint with supertest doesn't work well, and Jest keeps sending errors of unclosed connections. I am going to wait until V4 of Strapi is truly "stable" with proper test support.

[Maxservais]

@qunabu tried to do what you did in your repo but it doesn't seem to work for me since I use Typescript. Are you aware of any solution? Cheers mate!

[qunabu]

Typescript testing https://github.com/qunabu/strapi-unit-test-example/tree/ver4ts

[Maxservais]

@qunabu thank you so much mate, it's very helpful and got it to work thanks to you! Cheers

[github-actions]

Hello!

As we have not received any new or updated information to reproduce this issue in the last 14 days we are marking this issue as closed. Should you have new information please feel free to respond and we will consider reopening it.

If anyone else have updated information for this issue, please open up a new bug report and simply reference this closed bug report so that we can get any new information you may have. If you have questions please refer to the contributor's guide on opening issues.

Thank you and have a great day!

[douglaschalegre]

I'm having the same issue with the following code trying to use supertest and jest with typescript on v4 on 4.4.7
`
● process.exit called with "1"

   7 |   if (!instance) {
   8 |    
>  9 |     await Strapi().load();

`
The documentation doesn't show anything about unit testing with typescript, it's very sad that strapi says that supports typescript but always are new features that doesn't come with interfaces and others that simply doesn't work.

Also, how is this a bug with low priority? Unit testing is a very important part of CI/CD pipelines and in many cases cant even deploy without it... since strapi oficially suports typescript i think this should be ranked as a higher priority

[c00lc0de]

I am experiencing a similar issue. I have followed the official documentation, but I find the process to be excessively convoluted when trying to set up a basic configuration, including dummy data and programmatically setting permissions for testing endpoints, among other things.
Am seeing:

       6 | async function setupStrapi() {
       7 |   if (!instance) {
    >  8 |     await Strapi().load();
         |     ^
       9 |     instance = strapi;
      10 |
      11 |     await instance.server.mount();
      at Object.<anonymous>.module.exports [as bootstrap] (node_modules/@strapi/plugin-users-permissions/server/bootstrap/index.js:115:13)
      at Object.bootstrap (node_modules/@strapi/strapi/lib/core/domain/module/index.js:40:7)
      at Object.bootstrap (node_modules/@strapi/strapi/lib/core/registries/modules.js:28:9)
      at Strapi.runLifecyclesFunctions (node_modules/@strapi/strapi/lib/Strapi.js:544:5)
      at Strapi.bootstrap (node_modules/@strapi/strapi/lib/Strapi.js:482:5)
      at Strapi.load (node_modules/@strapi/strapi/lib/Strapi.js:491:5)
      at setupStrapi (src/tests/helpers/strapi.js:8:5)
      at Object.<anonymous> (src/tests/app.test.js:5:3)

For versions:

"devDependencies": {
    "@babel/eslint-parser": "~7.18.0",
    "eslint": "~8.16.0",
    "jest": "^29.5.0",
    "prettier": "~2.6.0",
    "sqlite3": "^5.1.6",
    "supertest": "^6.3.3"
  },
  "dependencies": {
    "@strapi/plugin-i18n": "~4.10.5",
    "@strapi/plugin-users-permissions": "~4.10.5",
    "@strapi/strapi": "~4.10.5",
    "mysql": "~2.18.1",
    "strapi-provider-upload-azure-storage": "~2.1.0"
  },

[c00lc0de]

Setting

"testTimeout": 30000

in the package.json did solve this issue for me.