Spike: migrate tests to Jest (ts-jest)

[bajtos]

Let's try to speed up our dev process and CI builds by migrating to ts-jest.

1. Jest can run tests in parallel and provides clever watch mode.
2. ts-jest compiles TypeScript files on the fly, removing the need for costly build.
3. Jest provides tools for snapshot-based testing, this would be very useful to test CLI code scaffolding TypeScript files and to test code emitting OpenAPI spec.

Known problems to resolve:

• @loopback/metadata relies on instanceof Function check that does not work under Jest, see fix(metadata): don't rely on instanceof checks #3160
• @loopback/boot is not able to find & load artifacts from original .ts files
• Tests using classes defined outside of the scope of test functions fail with ReferenceError, e.g. ReferenceError: Cannot access 'CustomerController' before initialization.
• Integration with express is broken
• Repository bindings are not picked up (The key 'repositories.NoteRepository' is not bound to any value in context application)

See the list of skipped files in https://github.com/strongloop/loopback-next/blob/test/jest/jest.config.js for the up-to-date information.

[bajtos]

Work started in https://github.com/strongloop/loopback-next/tree/test/jest

[mamiller93]

I'm curious whether this spike has an updated timeline as you removed the 2020Q1 tag about a month ago. From my research, a lot of users have expressed mocha is faster than jest, but we're also running into some slowness when running 300+ tests over 50+ files. I'm curious whether the migration to Jest would prove beneficial and whether we should embark on that adventure by ourselves or hold off and wait for the official loopback merge.

[bajtos]

@mamiller93 this spike is not a priority for us in the next few months (at least). The migration would require too many changes.

We are also discussing with @boneskull the option to improve Mocha to better serve our needs: allow the tests to be run in parallel, better support for TypeScript in watch mode.

I'm curious whether the migration to Jest would prove beneficial and whether we should embark on that adventure by ourselves or hold off and wait for the official loopback merge.

I guess the only way how to answer the first question is to do a spike on your side and see if you can get any speed up by moving to Jest.

Personally, I find Jest much better to use when working in test-driven way, as it has CLI UI allowing me to re-run only a subset of tests (e.g. only the tests that failed in the last run), update snapshots, etc. Having said that, I have only minimal experience with Jest, it's very likely I am not aware of many problems that we would discover if we migrated loopback-next to Jest.

[boneskull]

Alright, so the parallel mode is ready for testing by the LB team. Ref: mochajs/mocha#4198

Installation

1. In your loopback-next working copy root, navigate to packages/build

2. npm install -D https://github.com/mochajs/mocha#boneskull/issue/2839-concurrent (if you'd rather pin it to a changeset, the latest at the time of this writing is d0472ab).

3. Navigate back to your loopback-next working copy root.

4. Modify packages/build/config/.mocharc.json to add parallel: true:

   {
     "require": "source-map-support/register",
     "recursive": true,
     "exit": true,
     "reporter": "dot",
     "parallel": true
   }

5. Run npm test.

Regarding Test Results

• I am seeing a lot of MaxListenersExceededWarnings due to the Process.SIGTERM event being listened for, but the listeners are never removed.
• There are some unhandled promise rejections due to "socket hang up"
• There's an ENOENT for the uv_cwd syscall in internal/bootstrap/switches/does_own_process_state.js.
• Snapshots seem to be "missing." @raymondfeng mentioned something about a .sandbox directory which assumes tests are run in serial; this might be related and might need to change. Also related ENOENT errors.
• Further splitting up the generator tests into more files, if possible, will improve the speed of the test run; the tests slow down a bit at this point.
• I see 82 total failures.
• You can control the max number of worker processes via the jobs option. By default, it uses <number of CPU cores> - 1. Once this number of processes has been reached, test files will sit in a queue and will be picked up by worker processes as they complete test file runs. Once there are less tests remaining than the max number of worker processes, worker processes will remain open and idle until all tests are complete.

Notes

• I'm using Node.js v13.11.0.
• You need to use, at minimum, Node.js v10. Support for Node.js v8.x is already dropped in Mocha's master branch. If you need support for Node.js v8, we can discuss a backport.
• Test results will appear in "chunks"; the results of each test file are buffered, and output all at once when the file has completed.
• Use DEBUG=mocha:parallel* for more information about what's happening under the hood.
• Test files are executed across n worker processes, one file at a time.
• --file and --sort are unsupported.
• Doing anything with the root suite (e.g., top-level hooks) will only work within the context of a single file. I don't think LB is doing anything like this.
• --bail is best-effort.
• --watch is as-of-yet unsupported, but I plan to add it (maybe even to this PR).
• All built-in reporters should work.

Please let me know if you're stuck, and I can help out. Also would like to know when the team can carve out some time to look at this, since landing the PR will depend on your requirements.

Thanks!

[mdbetancourt]

in my project im running loopback with just typescript (ts-node, ts-node-dev) i changed booters to .ts and run ts-node with --type-check flag, and im running jest with ts-jest with no problems (until now)

[bajtos]

With #5011 in place, we are going to stick with Mocha for the foreseeable future.