test(client): add integration runner

[danstarns]

• Includes and closes: test/fix(client): await blocking IO #13169

Why

1. Slow Tests #13092
2. I was finding that the integration tests were running all at once and it was hard to tell jest to treat the integration differently without losing potential on the simple tests.

What

This PR excludes all of the tests in ./packages/client/tests/integration from the main test suite and introduces:

integration-runner.test.ts

This new file will readDir both the happy/ and errors/ tests and then perform sequential batching of n amount of tests.

Between each batch, a backoff is added so that we don't completely overwhelm other components such as the database.

How can I run them?

You should not need to change anything in your workflow, however, if you especially want to run the integration test's on their own you can run:

cd ./packages/client && pnpm run test integration

Findings

This shows my recorded stats of the time taken to run the client test suite before, with maxWorkers, and then after this addition.

Before

cd ./packages/client && pnpm run test

Could not run - Timed out!

Time:        560.228 s
Ran all test suites.
 ELIFECYCLE  Test failed. See above for more details.

cd ./packages/client && pnpm run test integration

Could not run - Timed out!

Time:        297.905 s
Ran all test suites matching /integration/i.
 ELIFECYCLE  Test failed. See above for more details.

Using maxWorkers

• test(client): add integration runner #13149 (review)

These run! However, the issue here is now that the entire test suite is running at 50% capacity when in reality we only want to limit the heavy load stuff(the integration tests).

cd ./packages/client && npx jest --maxWorkers="50%"

Pass

Time:        523.493 s, estimated 538 s
Ran all test suites.

cd ./packages/client && npx jest integration --maxWorkers="50%"

Pass

Time:        330.482 s
Ran all test suites matching /integration/i

After

cd ./packages/client && pnpm run test

Pass - 222.898 seconds faster

Time:        300.595 s
Ran all test suites.

cd ./packages/client && pnpm run test integration

Pass - 83.634 seconds faster

integration/integration-runner
    √ happy (243661 ms)
    √ errors

Test Suites: 1 passed, 1 total
Tests:       2 passed, 2 total
Snapshots:   0 total
Time:        246.848 s
Ran all test suites matching /integration/i.

Regressions

Some things we miss out on in this implementation, this is because we are now spawning Jest using exec. This means that the main Jest process doesn't have a full collective view of how many tests are run for example or the coverage of those files tested on inside the spawned process.

1. Jest Stats - You can still see all in the console but it won't be in one nice neat place like normal.
2. Code Coverage? - IDK yet standby

[aqrln]

Have you tried limiting maxWorkers in jest.config.js? This is not the same thing as what you're doing in this PR (which is also an interesting idea), but it might've solved the issue.

[danstarns]

Have you tried limiting maxWorkers in jest.config.js? This is not the same thing as what you're doing in this PR (which is also an interesting idea), but it might've solved the issue.

Yes, and this does help. I have added a section in the PR description Findings that shares my stats using maxWorkers.

[danstarns]

If this process errors the entire suite will fail fast, we could catch the error and propagate it like with the logs below. WDYT?

[danstarns]

Logging all at once so that it eases the load on your terminal and that they are all displayed one after the other. We might want to log errors here too...