Jest support config as function

[jimCresswell]

Question

Hi. Is anyone aware of configuration or a plugin that would allow StrykerJS to work with a Next.js app using Typescript and the now default SWC compiler rather than Babel?

I feel that using the Babel plugin wouldn't be appropriate, given it isn't used to transpile the tsx files for the app.

I'd love to get this working, the app has Node, browser and lambda code, and while the test coverage is going up we need a way to validate the worth of that coverage.

If this needs a new plugin maybe someone @vercel could help, it would certainly benefit them.

Thanks in advance

[nicojs]

Hi @jimCresswell, excellent question.

Actually, Stryker doesn't care about which build command you use to build your project. It does however use Babel to perform the actual mutations. Are you running into specific errors?

[jimCresswell]

Hi @nicojs, okay that's really interesting, I hadn't realised Babel was fundamental to Stryker. I've attached the console output from invoking StrykerJS with npm run stryker run, and the config (default + Typescript) looks like

/**
 * @type {import('@stryker-mutator/api/core').StrykerOptions}
 */
module.exports = {
  _comment:
    "This config was generated using 'stryker init'. Please take a look at: https://stryker-mutator.io/docs/stryker-js/configuration/ for more information",
  packageManager: "npm",
  reporters: ["html", "clear-text", "progress"],
  testRunner: "jest",
  coverageAnalysis: "perTest",
  checkers: ["typescript"],
  tsconfigFile: "tsconfig.json",
};

The errors look like

SyntaxError: /home/jim/code/oak/samara/.stryker-tmp/sandbox6144601/src/__tests__/pages/index.test.tsx: Support for the experimental syntax 'jsx' isn't currently enabled (10:25):

   8 | describe("pages/index.tsx", () => {
   9 |   it("Renders lesson title ", async () => {
> 10 |     renderWithProviders(<Home lesson={testLesson} />);

and

Details:

/home/jim/code/oak/samara/.stryker-tmp/sandbox6144601/src/__tests__/__helpers__/apolloMocks.ts:2
import { BookmarkedLessonsDocument, BookmarkedLessonAddDocument, BookmarkedLessonRemoveDocument } from "../../browser-lib/graphql/generated/apollo";
^^^^^^

SyntaxError: Cannot use import statement outside a module

and

> 10 |   signInWithEmail: async (email: string) => undefined,
     |                                ^
  11 |   signInWithEmailCallback: async () => undefined,
  12 | };
  13 | SyntaxError: /home/jim/code/oak/samara/.stryker-tmp/sandbox6144601/src/__tests__/__helpers__/MockedAuthProvider.tsx: Unexpected token, expected "," (10:31)

The first two types I understand, the project has no Babel config, so Stryker doesn't attempt to use Babel. Does that explain not understanding the Typescript syntax as well?

Would you recommend I manually manage the Stryker plugins to include Babel then?

Do you have any feeling of the scope of porting the Babel powered mutations to work with SWC? Maybe it just isn't necessary, but I think it's worth asking the question.

[nicojs]

Do you have any feeling of the scope of porting the Babel powered mutations to work with SWC? Maybe it just isn't necessary, but I think it's worth asking the question.

That's unnecessary. What Stryker will do is mutate your source files and replace them with source files containing mutants. Users shouldn't worry too much about which library Stryker uses underneath to accomplish this. Babel is perfect for this because it supports all JS and friends syntax AND has a high-level API for manipulating the AST

By the look of it, this error is reported by Jest right after Stryker has mutated the code. This means that Stryker worked, you can open up a source file inside the sandbox6144601 directory to make sure: the code should contain mutants. F.y.i. the sandbox directory gets deleted automatically after a successful Stryker run.

So: Jest has issues. Could you try to run jest normally? So does this command work: npx jest?

If that command works as expected: does this command work from the sandbox directory? node ../../node_modules/jest/bin/jest.js

[jimCresswell]

Ahh, both of those commands succeed! Our Jest config uses the Next.js Jest config, like

// jest.config.js
const nextJest = require("next/jest");

const createJestConfig = nextJest({
  // Provide the path to your Next.js app to load next.config.js and .env files in your test environment
  dir: "./",
});
...

I wonder if the invocation inside of Stryker is somehow not picking up that config.

Thanks for all your help by the way, this is incredibly useful.

[nicojs]

Ah good to know!

Then there is a problem with the jest-runner Stryker plugin. Could you try to explicitly specify the jest config file?

{
  "jest": {
    "projectType": "custom",
    "configFile": "jest.config.js"
  }
}

See https://stryker-mutator.io/docs/stryker-js/jest-runner#configuration for more info.

You can run Stryker with --logLevel debug to see exactly which configuration file is read by the jest-runner plugin.

Thanks for all your help by the way, this is incredibly useful.

No problem, I'm hoping to improve the user experience for newcomers.

[jimCresswell]

Okay! I've run that with updated config

const path = require("path");

/**
 * @type {import('@stryker-mutator/api/core').StrykerOptions}
 */
module.exports = {
  _comment:
    "This config was generated using 'stryker init'. Please take a look at: https://stryker-mutator.io/docs/stryker-js/configuration/ for more information",
  packageManager: "npm",
  reporters: ["html", "clear-text", "progress"],
  testRunner: "jest",
  jest: {
    projectType: "custom",
    configFile: path.resolve(__dirname, "jest.config.js"),
  },
  coverageAnalysis: "perTest",
  checkers: ["typescript"],
  tsconfigFile: "tsconfig.json",
};

I've attached the new log with debug output included stryker.log.

I think line 769 shows it isn't reading the Jest config, and I wonder if it's because it's returned as an async function. That's supported by Jest, and necessary for running with the Next.js config because that is also async, but the StrykerJS code looks like it assumes a simple object export?

[nicojs]

Ah yes, an async function is probably not supported!

Final jest config: null

☝ this is the smoking gun!

What exactly is your jest.config.js file exporting? A promise that resolves an object? Or an async function?

It should be a pretty simple fix, we should await any configuration:

stryker-js/packages/jest-runner/src/config-loaders/custom-jest-config-loader.ts

Line 23 in dce5882

public loadConfig(): Config.InitialOptions {

[jimCresswell]

Good news! Feels like it should work.

It's an async function, like this example in the Jest docs

// Or async function
module.exports = async () => {
  return {
    verbose: true,
  };
};

Specifically the result of calling this function https://github.com/vercel/next.js/blob/v12.1.4/packages/next/build/jest/jest.ts#L43 like

nextJest({ dir: "./" })(customJestConfig)

[jimCresswell]

@nicojs I can have a go at a PR next week, or I'm happy to leave it with you?

[nicojs]

A PR would be much appreciated! I've planned 2 days to work on Stryker next week and the week after, so I will have the time to review it.

[jimCresswell]

No problem, I will do my best! I am travelling next week so I will either have some spare time or no internet, we'll see.

[jimCresswell]

I had a quick go at this, changing the config loader seems fairly straightforward (needed a new interface)

export class CustomJestConfigLoader implements JestConfigLoaderAsync {
  public static inject = tokens(commonTokens.logger, commonTokens.options, pluginTokens.requireFromCwd);

  constructor(private readonly log: Logger, private readonly options: StrykerOptions, private readonly requireFromCwd: typeof requireResolve) {}

  public async loadConfig(): Promise<Config.InitialOptions> {
    const jestConfig = (await this.readConfigFromJestConfigFile()) ?? this.readConfigFromPackageJson() ?? {};
    this.log.debug('Final jest config: %s', jestConfig);
    return jestConfig;
  }

  private async readConfigFromJestConfigFile(): Promise<Config.InitialOptions | undefined> {
    const configFilePath = this.resolveJestConfigFilePath();
    if (configFilePath) {
      const potentialConfig: unknown = this.requireFromCwd(configFilePath);
      let config;
      if (typeof potentialConfig === 'function') {
        config = (await potentialConfig()) as Config.InitialOptions;
      } else {
        config = potentialConfig as Config.InitialOptions;
      }
      this.log.debug(`Read Jest config from ${configFilePath}`);
      this.setRootDir(config, configFilePath);
      return config;
    }
    return undefined;
  }

but I couldn't at first glance see how to update the unit tests, or validate that the consuming code could cope with loadConfig becoming async. Next week is looking pretty full atm, would you have time to look at this? If not I will try to get to it post-travel.

[juliusmarminge]

@jimCresswell did you get any workaround to work? I'm trying to setup stryker for my next.js project too but so far no luck

[jimCresswell]

@juliusmarminge no, if you are using the Next Jest set up then you will need to wait for the above fix or something like it to go in. You could switch to a custom Jest set up, but you would have to return the config synchronously, which necessarily means you couldn't read in the Next config

[juliusmarminge]

@jimCresswell alright, I'll wait for the fix!.

One workaround I found is to setup a .babelrc with some presets, and use custom test runner:

// .babelrc.js
module.exports = {
  presets: ["@babel/preset-react", "@babel/preset-typescript"],
  plugins: [],
};

// stryker.config.js
module.exports = {
  _comment:
    "This config was generated using 'stryker init'. Please take a look at: https://stryker-mutator.io/docs/stryker-js/configuration/ for more information",
  packageManager: "npm",
  reporters: ["html", "clear-text", "progress"],
  testRunner: "command",
  coverageAnalysis: "perTest",
};

But you would have to remove the babel-config when you build your project so that next uses SWC...

Stryker output with that config:

stryker run
10:57:46 (14141) INFO ConfigReader Using stryker.conf.js
10:57:48 (14141) INFO InputFileResolver Found 50 of 509 file(s) to be mutated.
10:57:49 (14141) INFO Instrumenter Instrumented 50 source file(s) with 2647 mutant(s)
10:57:49 (14141) INFO ConcurrencyTokenProvider Creating 7 test runner process(es).
10:57:50 (14141) INFO DryRunExecutor Starting initial test run (command test runner with "perTest" coverage analysis). This may take a while.
10:57:52 (14141) INFO DryRunExecutor Initial test run succeeded. Ran 1 tests in 2 seconds (net 2289 ms, overhead 0 ms).
Mutation testing  [] 3% (elapsed: <1m, remaining: ~19m) 98/2647 tested (98 survived, 0 timed out)

[jimCresswell]

Yes, that would work, but as you say it's a bit involved 😃

[nicojs]

@jimCresswell your suggestion in #3480 (comment) is exactly what I meant. Would you feel comfortable implementing it in a PR?

[jimCresswell]

Hi @nicojs, sorry for the delay. Sure I can raise a PR, but I might need some help sorting out the unit tests to cover the async change. Also probably next week at the earliest.

[jimCresswell]

I won't have time to work on this for a while. If it's still an issue when I'm available I'll pick it up again.

[nicojs]

Sure, no problem. I've assigned the Good first issue label, maybe that helps

[NerOcrO]

Anyone for this PR?
It will be very usefull for me :)
May be there is a tricks in the mean time?

[nicojs]

@NerOcrO PRs are still welcome, feel free to take a shot at it 🙇‍♂️

I recently ran into this issue as well and used this dirty workaround. I use the jest command to run normal unit test, and write the awaited config to a jest.json file. Then later during mutation testing, I use that json file.

Example jest config

const nextJest = require('next/jest');

const createJestConfig = nextJest({
  // Provide the path to your Next.js app to load next.config.js and .env files in your test environment
  dir: './',
});

// Add any custom config to be passed to Jest
const customJestConfig = {
  setupFilesAfterEnv: ['<rootDir>/jest.setup.ts'],
  moduleNameMapper: {
    // Handle module aliases (this will be automatically configured for you soon)
    '^@/components/(.*)$': '<rootDir>/components/$1',

    '^@/pages/(.*)$': '<rootDir>/pages/$1',
  },
  extensionsToTreatAsEsm: ['.ts', '.tsx'],
  testEnvironment: 'jest-environment-jsdom',
};

// createJestConfig is exported this way to ensure that next/jest can load the Next.js config which is async
module.exports = async () => {
    const config = await createJestConfig(customJestConfig)()
+   require('fs').writeFileSync('jest.json', JSON.stringify(config, null, 2));
    return config;
};

Example StrykerJS config

{
  "$schema": "./node_modules/@stryker-mutator/core/schema/stryker-schema.json",
  "_comment": "This config was generated using 'stryker init'. Please take a look at: https://stryker-mutator.io/docs/stryker-js/configuration/ for more information",
  "packageManager": "npm",
  "reporters": [
    "html",
    "clear-text",
    "progress"
  ],
  "testRunner": "jest",
  "coverageAnalysis": "perTest",
+ "jest": {
+   "configFile": "jest.json"
+ },
  "concurrency": 2
}

[NerOcrO]

Hi @nicojs
Thx for your response!
Unfortunately, it doesn't work for me:

ERROR Stryker Unexpected error occurred while running Stryker StrykerError: Error: Could not inject [class ChildProcessTestRunnerWorker] -> [function jestTestRunnerFactory] -> [class JestTestRunner].
Cause: Cannot find module '/.../app/.stryker-tmp/sandbox761584/jest.json'

[nicojs]

Can you confirm the jest.json file exists before running Stryker?

[NerOcrO]

Oh ok! Now it works!
thx :)

[nicojs]

Note: we are working on getting this functionality exported from jest-config so we can support it out-of-the-box.

[jimCresswell]

That's fantastic, and greatly appreciated.

[nicojs]

Closed with #3761. We will release it soon.

[jimCresswell]