JEST: Improve Jest Support

[matthewmueller]

Problem

The user describes it well in the issue:

Currently trying to migrate to the library engine for an existing project (>1kloc prisma schema). After enabling the nApi flag, I've noticed that Jest will periodically crash while running test suites serially (hundreds of tests spread across almost 30 test suites).

After taking a closer look using jest's --logHeapUsage flag, I've found that memory usage goes up by 100MB+ per test suite. About thirds into a test run, jest would be eating up over 2GB of memory and crash soon after.

Goal

Any steps we can do to improve the memory usage would be helpful for our community.

If we can't fully solve the issue, let's approach this incrementally: allow community to first run 500 tests serially, check-in with the community, then we'll consider 1000 and so on as we learn people are still having problems.

Next Steps

• JEST: Breakdown into next steps #12370

Issues

This issue collects issues related to Jest support with Prisma that might be related or closable after we resolve the problem described above.

Memory

• Memory leaks in Jest when running tests serially with nApi enabled #8989
• test(client/debug): switch from ts-jest to swc #12242
• test: reproduce "minimal" Jest memory leak #11762
• fix: Memory leaks in Jest when running tests serially with nApi enabled #11751

Flakiness

• Flaky Jest tests on GH Actions: ERROR  Command failed with exit code 139. Segmentation fault (core dumped) #8214

Mocking

• Attempting to mock clocks causes prisma create to never return #7424

Internal Usage of Jest

• Cannot upgrade some dependencies to their latest version because they are ESM only  #7618

[matthewmueller]

@millsp mentioned that we might need to do a breaking change for the WASM module integration to continue supporting something like Prisma.dmmf.

We'll need to decide if this is worth maintaining until P4 or it's okay to break earlier.

[aqrln]

Related upstream issue: jestjs/jest#11956

[m-allanson]

Commenting here as I think there's more issues related to this.

The problem

This is my understanding of the problem, I'd love to be corrected if I've got it wrong!

• There's maybe a memory leak between Prisma Client and Prisma Engine when creating many instances of Prisma Client.
• Typically you run Prisma Client as a singleton, so leaks don't happen or don't cause a problem.
• In a testing environment you'll often create many Prisma Client instances which makes leaks more obvious.
• There's maybe some issues in Jest that exacerbate leaks in Prisma. And potentially some recent changes in Node that also make the problem more obvious.

Does the above sound correct?

Related issues

There are a lot of open issues related to this problem. As well as the issues listed above, I found the following:

• Jest spawns quite a many binaries
• Slow Tests
• OOM: Out of memory with library engine with Node.js v16 in test suite
• Node-API/LibraryEngine: $disconnect does not free up memory / kill engine
• Prisma memory leaks on Multitenant applications
• Add $destroy to the client so it cannot be reused
• Prisma client does not clean up itself after test

Maybe also #12152 would help to lock down instances of this problem.

Workarounds

When it comes to Jest, the only workaround so far is to run less tests.

It'd be amazing if something like #12153 could be implemented as part of this work. As the memory problem seems to be affecting a lot of people and doesn't have a clear fix or workaround yet.

[knsg16]

@m-allanson Thanks for summarizing the situation. Really appreciated.
We have the same heap problem when running jest with Prisma on Github Actions.

[m-allanson]

There's a workaround for this using jest@28's new test sharding feature. Each test shard can run as a new process, so you can divide your tests into as many shards as you need to avoid running out of memory.

As a demo I forked @aqrln's leak example to use sharding: m-allanson/prisma-leak-repro@6e5e273.

If your tests are running on GitHub Actions, you can also use their matrix feature to run the shards in parallel.

[m-allanson]

The new prisma@4 works much better with Jest. Here's two runs based on @aqrln's prisma-leak-repro. One using Prisma v4, and one using v3.

The main difference:

• With Prisma v4 the tests run in ~22 seconds with a max 963MB heap size
• With Prisma v3 the tests take ~48 seconds to get half way, then explode at a 4003MB heap size

The v4 tests run faster with much less memory usage 🙌 Thanks Prisma folks who worked on this :)

Prisma v4

❯ npm run test

> simple-db@1.0.0 test
> make test

node --expose-gc ./node_modules/.bin/jest --runInBand --logHeapUsage --no-cache tests-generated
 PASS  tests-generated/32.test.ts (389 MB heap size)
 PASS  tests-generated/22.test.ts (404 MB heap size)
 PASS  tests-generated/50.test.ts (416 MB heap size)
 PASS  tests-generated/40.test.ts (428 MB heap size)
 PASS  tests-generated/49.test.ts (440 MB heap size)
 PASS  tests-generated/14.test.ts (452 MB heap size)
 PASS  tests-generated/7.test.ts (463 MB heap size)
 PASS  tests-generated/48.test.ts (475 MB heap size)
 PASS  tests-generated/6.test.ts (487 MB heap size)
 PASS  tests-generated/15.test.ts (499 MB heap size)
 PASS  tests-generated/23.test.ts (510 MB heap size)
 PASS  tests-generated/33.test.ts (522 MB heap size)
 PASS  tests-generated/41.test.ts (534 MB heap size)
 PASS  tests-generated/28.test.ts (545 MB heap size)
 PASS  tests-generated/38.test.ts (557 MB heap size)
 PASS  tests-generated/17.test.ts (569 MB heap size)
 PASS  tests-generated/4.test.ts (580 MB heap size)
 PASS  tests-generated/43.test.ts (592 MB heap size)
 PASS  tests-generated/31.test.ts (604 MB heap size)
 PASS  tests-generated/21.test.ts (615 MB heap size)
 PASS  tests-generated/42.test.ts (627 MB heap size)
 PASS  tests-generated/20.test.ts (638 MB heap size)
 PASS  tests-generated/30.test.ts (650 MB heap size)
 PASS  tests-generated/39.test.ts (662 MB heap size)
 PASS  tests-generated/29.test.ts (673 MB heap size)
 PASS  tests-generated/16.test.ts (685 MB heap size)
 PASS  tests-generated/13.test.ts (696 MB heap size)
 PASS  tests-generated/5.test.ts (708 MB heap size)
 PASS  tests-generated/47.test.ts (720 MB heap size)
 PASS  tests-generated/9.test.ts (731 MB heap size)
 PASS  tests-generated/35.test.ts (743 MB heap size)
 PASS  tests-generated/8.test.ts (755 MB heap size)
 PASS  tests-generated/25.test.ts (766 MB heap size)
 PASS  tests-generated/46.test.ts (778 MB heap size)
 PASS  tests-generated/24.test.ts (789 MB heap size)
 PASS  tests-generated/34.test.ts (801 MB heap size)
 PASS  tests-generated/12.test.ts (813 MB heap size)
 PASS  tests-generated/1.test.ts (824 MB heap size)
 PASS  tests-generated/19.test.ts (836 MB heap size)
 PASS  tests-generated/36.test.ts (847 MB heap size)
 PASS  tests-generated/26.test.ts (859 MB heap size)
 PASS  tests-generated/44.test.ts (871 MB heap size)
 PASS  tests-generated/3.test.ts (882 MB heap size)
 PASS  tests-generated/10.test.ts (894 MB heap size)
 PASS  tests-generated/2.test.ts (905 MB heap size)
 PASS  tests-generated/11.test.ts (917 MB heap size)
 PASS  tests-generated/18.test.ts (928 MB heap size)
 PASS  tests-generated/27.test.ts (940 MB heap size)
 PASS  tests-generated/37.test.ts (951 MB heap size)
 PASS  tests-generated/45.test.ts (963 MB heap size)

Test Suites: 50 passed, 50 total
Tests:       50 passed, 50 total
Snapshots:   0 total
Time:        21.643 s
Ran all test suites matching /tests-generated/i.

Prisma v3

❯ npm run test

> simple-db@1.0.0 test
> make test

node --expose-gc ./node_modules/.bin/jest --runInBand --logHeapUsage --no-cache tests-generated
 PASS  tests-generated/32.test.ts (555 MB heap size)
 PASS  tests-generated/22.test.ts (712 MB heap size)
 PASS  tests-generated/49.test.ts (835 MB heap size)
 PASS  tests-generated/50.test.ts (957 MB heap size)
 PASS  tests-generated/40.test.ts (1079 MB heap size)
 PASS  tests-generated/7.test.ts (1201 MB heap size)
 PASS  tests-generated/14.test.ts (1323 MB heap size)
 PASS  tests-generated/6.test.ts (1445 MB heap size)
 PASS  tests-generated/48.test.ts (1567 MB heap size)
 PASS  tests-generated/15.test.ts (1689 MB heap size)
 PASS  tests-generated/23.test.ts (1811 MB heap size)
 PASS  tests-generated/33.test.ts (1933 MB heap size)
 PASS  tests-generated/41.test.ts (2055 MB heap size)
 PASS  tests-generated/28.test.ts (2177 MB heap size)
 PASS  tests-generated/17.test.ts (2299 MB heap size)
 PASS  tests-generated/38.test.ts (2421 MB heap size)
 PASS  tests-generated/4.test.ts (2542 MB heap size)
 PASS  tests-generated/21.test.ts (2664 MB heap size)
 PASS  tests-generated/43.test.ts (2786 MB heap size)
 PASS  tests-generated/31.test.ts (2908 MB heap size)
 PASS  tests-generated/42.test.ts (3030 MB heap size)
 PASS  tests-generated/20.test.ts (3152 MB heap size)
 PASS  tests-generated/16.test.ts (3274 MB heap size)
 PASS  tests-generated/30.test.ts (3395 MB heap size)
 PASS  tests-generated/39.test.ts (3517 MB heap size)
 PASS  tests-generated/29.test.ts (3639 MB heap size)
 PASS  tests-generated/5.test.ts (3761 MB heap size)
 PASS  tests-generated/13.test.ts (3883 MB heap size)
 PASS  tests-generated/9.test.ts (4003 MB heap size)

 RUNS  tests-generated/35.test.ts

<--- Last few GCs --->

[32438:0x128008000]    44972 ms: Scavenge 4015.3 (4126.0) -> 4010.0 (4141.8) MB, 5.4 / 0.0 ms  (average mu = 0.329, current mu = 0.239) allocation failure
[32438:0x128008000]    46791 ms: Mark-sweep (reduce) 4042.9 (4174.7) -> 4003.5 (4097.5) MB, 1602.9 / 0.1 ms  (+ 3.6 ms in 1 steps since start of marking, biggest step 3.6 ms, walltime since start of marking 1615 ms) (average mu = 0.234, current mu = 0.156

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

make: *** [test] Abort trap: 6
Command exited after 48 seconds

[Jolg42]

@m-allanson Good to hear! In Prisma 4 the main change here was done in #13811

From release notes https://github.com/prisma/prisma/releases/tag/4.0.0

Removal of the internal schema property from the generated Prisma Client

We've removed the internal Prisma.dmmf.schema to reduce the size of Prisma Client generated and improve boot times.

To access the schema property, you can use the getDmmf() method from @prisma/internals.

[millsp]

@m-allanson We indeed have alleviated some of the symptoms, I'm very happy that this runs better for you.
However, you might notice that the heap still grows without being freed, this means we still have work to do :)

[janpio]

I am closing this issue now as the original linked issues have been closed for a while now.

There are still some memory problems, also and especially when using with Jest. We have issues tracking them:

• label topic: performance/memory
• label topic: memory leak
• label topic: jest

If you still have problems with Prisma and Jest, please see if an appropriate issues for that already exists under these labels - or create a new one an we'll figure it out. Thank you!