Add property based testing based on fast-check in Jest

[dubzzz]

🚀 Feature Proposal

Property based testing can be seen as some kind of fuzzing approach. It makes users able to cover a wider range of inputs with a single test.

Motivation

fast-check proved very useful on Jest and was behind the following issues:

• toEqual not symmetric for Set #7975
• toStrictEqual fails to distinguish 0 from 5e-324 #7941
• toStrictEqual does not consider arrays with objects having undefined values correctly #7937 (for this one, the tests of fast-check failed because of the regression so I started to dig a bit and run property based on expect of Jest to see if I would be able to find other bugs like this one)

It is currently being added in the test suites of Jest itself: #8012

Example

I think the integration of fast-check within Jest could be something like the one I did for ava: https://github.com/dubzzz/ava-fast-check/ but I am very opened to discussions on this point.

My current wrapper works as follow:

• It provides testProp - and certainly itProp for Jest
• Its signature is: testProp(testName, generators, property)
• What's great with this wrapper is that it adds the seed next to the test name so that the user can have access to the seed even in case the test goes into an infinite loop (if and only if Jest can stop it in this case).

// for all a, b, c strings
// b is a substring of a + b + c
testProp('should detect the substring', [fc.string(), fc.string(), fc.string()], (a, b, c) => {
  return (a + b + c).includes(b);
});

or maybe:

// for all a, b, c strings
// b is a substring of a + b + c
test.prop('should detect the substring', [fc.string(), fc.string(), fc.string()], (a, b, c) => {
  return (a + b + c).includes(b);
});

Pitch

Jest is a very successful testing framework.
Adding property based testing directly within Jest would give the community a new way to check their code. It might help the JavaScript community to detect new bugs they even not though about before. fast-check has already been useful in many projects, see https://github.com/dubzzz/fast-check/blob/master/documentation/IssuesDiscovered.md

[dubzzz]

Something useful for the integration: fast-check follows semantic versioning.

Patch version should not:

• modify the values that would be generated by an arbitrary for a given seed - Except maybe if there is a crash or infinite loop in the generation part
• impact the shrinking tree
• impact replay - the same seed and path would lead to the same value - direct consequence of the two points above
• add new features

A minor should not:

• break existing APIs
• remove depreciated APIs

[dubzzz]

I am obviously a little biased concerning the choice of the framework as I wrote fast-check. In a nutshell when I first wanted to do property based in my company I opted for jsverify but got stuck on some of its limitations (due to initial design choices). So I started to explore how it worked and somehow started fast-check.

Why fast-check?
Maintained, used by awslabs/aws-cdk for instance, multiple unique features: replay, verbose, commands for UI tests, pre-conditions...
More on the read me.

See #7938 (comment)

[jeysal]

Having recently discovered #8032, just wanted to note that if we integrate something like this, we should definitely do it in a way that does not impact performance for users who do not use this feature (i.e. not unconditionally require('fast-check') from jest-circus or something).
In general I think it would be cleaner to have this in a standalone package, along the lines of ava-fast-check. @dubzzz do you think there would be any technical difficulties with doing that or other reasons why integrating more tightly with Jest would be beneficial?

[spontoreau]

Great idea!

[dubzzz]

Initially, I was not planning to add fast-check as an internal feature of Jest. But when @SimenB suggested it I really thought about this idea: I personally do think it would be a great thing for testing in JavaScript.

Indeed Jest has always proved ahead of time concerning test methodologies. For instance, when it came with snapshot tests, many developers tried and adopted it.

In a way, fast-check would be able to live alone without being directly integrated as a dependency of Jest. Doing a jest-fast-check would not be a big deal on my side and I can do it without any problems.

But I believe that pushing the property based testing through Jest, as one of its features, would highly increase the usage of the technic among dev communities. The more people will use such methodologies the more they will be confident about their code.

[jeysal]

I'm not totally against adding something like this and I'd like to hear what others think about this. Just wanted to note that if there's no technical reason why it should be in Jest core, we could also consider promoting it in other ways (documentation, jest-community, etc.).

[dubzzz]

Totally agree, there is no technical reason to move it inside Jest core. Promoting the approach through documentation or Jest community might also be great. I let you see what's the best option ;)

[quasicomputational]

Potentially, Jest could tell the fuzzer to do more/less work depending on how much other work there is to do, whether it's watch mode or one-shot, and probably other considerations. But that would be a nice-to-have and probably need a deeper integration than is being considered.

[dubzzz]

Jest could tell the fuzzer to do more/less work [...] watch mode or one-shot

I really like the idea suggested by @quasicomputational. Indeed by default, property based tends to take longer than classical units because it runs 100 tests for each property.

If fast-check was integrated as a dependency of Jest, such feature would be easy to build. fast-check already accepts options telling it to do only N runs (instead of 100 which is the default).

Syntax: fc.assert(/* your property */, { numRuns: 10, /* other settings */ })

If not, I don't know if there is a simple and long-term way to know whether the test is running in watch mode.

[dubzzz]

Another feature that would be more difficult to integrate if fast-check is not directly part of Jest is the retryTimes - more

By default, what you would want to retry in the case of property based is not the test itself but the execution of one run of the property.

[dubzzz]

@SimenB Any opinions on this RFC?

[quasicomputational]

Another benefit to having Jest know about the property tests: being able to isolate iterations better, including resetting mocks, timers and modules. This is particularly important to get right for shrinking, because if you've managed to contaminate the environment so that the property always fails, it'll be shrunk away to nothing and you get a useless report.

[migueloller]

testcheck seems to have support for Jest (although maybe a tad outdated). And regardless of the integration it can still be used without it. I've had success using testcheck with Jest in the past. How would you compare testcheck with fast-check?

[dubzzz]

Today, fast-check can easily be integrated into Jest without specific tooling.

test('here is a jest test using fast-check imported as fc', () => {
  fc.assert(
    fc.property(
      fc.nat(), fc.nat(),
      (a, b) => expect(a + b).toBe(b + a)
    )
  );
});

The benefits of adding such technology directly into Jest will be:

• more visibility on the approach
• better integration in watch mode: smaller number of tests for watch mode, replay of the last failure if any...
• run beforeEach, afterEach for each run of the predicate (today it requires manual wiring in fast-check)

Concerning benefits of fast-check over others. Today I would say shrinking capabilities go further (I can provide you example in private if you need to), replay options, verbose mode, pre-condition checks...

[migueloller]

testcheck can also be easily integrated without specific tooling. Here's an example of a test in our source code:

  it('normalizes the spans to strictly positive integers', () => {
    check(
      property(partGen, part => {
        normalize(part).forEach(span => {
          expect(span).toBeGreaterThan(0)
          expect(Number.isInteger(span)).toBe(true)
        })
      }),
    )
  })

The reason I bring up alternatives is because the Jest API is already quite extensive and I wonder if adding property testing to the matchers is something that should be left to third-party libraries 🤔

I think it would definitely be useful as part of the built-in matchers. The question is if it should be built from scratch with a new API considering Jest's philosophy or if a third-party library should be embedded into it. I'd love to hear what the Jest core team's thoughts are on this.

[stephen-smith]

Can you interleave properties/generation around describe/it? For example:

describe("MyClass instances", () => {
  fc.property(myClassGen, myInstance => {
    it("has foo", () => {
      expect(myInstance.foo()).toBe(true)
    });
    it("doesn't bar", () => {
      expect(myInstance.bar()).toBe(false)
    });
    it("munges any YourClass", () => {
      fc.property(yourClassGen, yourInstance => {
        expect(myInstance.munge(yourInstance));
      });
    });
  });
});

Sometimes I want generator to be run for a "suite" rather than for each individual test, but only sometimes.

[munizart]

@dubzzz having a test.forAll with similar functionality as test.each and prop matchers would be another nice feature (both in the case of a jest-fast-check lib or integrating into jest)

examples:

test('sum is associative', () => {
  const sum = (a, b) => a + b
  // calling sum with nat should always return another nat
  expect(sum).toBeClosed([fc.nat(), fc.nat()], fc.nat())
})

[dubzzz]

If you are interested, I am working on a package to ease integration of fast-check with jest. See https://github.com/dubzzz/jest-fast-check

All suggestions are welcomed (marchers, watch options for easier and faster replays). Meanwhile, having fast-check as a first class citizen would be even better 😊

[github-actions]

This issue is stale because it has been open for 1 year with no activity. Remove stale label or comment or this will be closed in 14 days.

[dubzzz]

Actually already done, closing it!

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.

[SimenB]

@dubzzz there's an incoming seed CLI argument via #12922 - is this something you could use for seeding in fast-check? That PR doesn't expose it (and fails if normalize isn't provided), but that's an easy fix if it'd be useful exposed on e.g. jest.getSeed() or some such.

Also, I'd be happy to link to @fast-check/jest in the docs somewhere.

[dubzzz]

Whaou such a good news. If I can access it I could definitely leverage it for @fast-check/jest. Is there any way to access it from @jest/globals or will there be any?

[SimenB]

I'm thinking of just putting it on jest.getSeed() or something if it's useful to you as is 🙂

So yeah

import {jest} from '@jest/globals';

console.log(jest.getSeed());

might need a better name?

[dubzzz]

What's the format for the seed? Double value? Int32? Uint32? Or maybe no strong guarantee?

jest.getSeed() would probably be good (can I get jest global from @jest/globals? No strong issue not to have it, mostly asking)

[SimenB]

What's the format for the seed? Double value? Int32? Uint32? Or maybe no strong guarantee?

https://github.com/facebook/jest/pull/12922/files#diff-9dbb1da56fa2045712eef8db063b52b3555a799b93e858b7449b5561bef52d9fR1143-R1147

If link is wonky - current implementation is whatever the user passed (which we should validate to ensure it conforms to our contract) or generating one in the shape of Math.floor((1 - Math.random()) * (Math.pow(2, 29) - 1))

[dubzzz]

Perfect 🥰 I'll draft something into @fast-check/jest soon to leverage it, thank you so much for it @SimenB

[SimenB]

note that it hasn't landed it, but I hope to do so in a week or two (I'm heading on vacation tomorrow, but will probably find some computer time in the week I'm gone), but exciting to hear it'll work for your use case 👍

[dubzzz]

You'll probably have the answer quickly: is there a way to get the currently registered beforeEach out of jest? Or even better manually trigger it?

Why would I need that for fast-check? Because I need to rerun beforeEach and afterEach before and after any execution of the predicate. So being able to either access the registered one or trigger it would help.

[dubzzz]

@SimenB Not sure it's the case in the current 'seed' implementation but do you display the seed in case of failure? When manually defined it's questionable but seems to be needed in the automatic seed case.

[SimenB]

is there a way to get the currently registered beforeEach out of jest?

I don't think so - closest might be handleTestEvent?

Not sure it's the case in the current 'seed' implementation but do you display the seed in case of failure?

Not decided. My current thinking is just landing a way of setting a seed, and it's up to the thing that uses this seed to print it

[dubzzz]

Thanks for all these answers, I'll check them as soon as the version of Jest get released in order to improve as much as possible @fast-check/jest. But definitely the new seed and showSeed options will help a lot.

By the way, I'm planning to change the API of @fast-check/jest and make it even closer to the real one provided by Jest. Up-to-now it used to expose two functions (with extra options like .concurrent and co as Jest does) called itProp and testProp (see documentation). In the next major I plan to make the API closer than the one of Jest by providing a revamped version of it and test. This revamped version will come with a new extra option called .prop and can be seen as a .each but for Property Based Testing.

Not fully made my mind about how I'll do but here is the idea: 3303

It's probably a good target to have a very close API in @fast-check/jest if one day fast-check and .prop want to be part of Jest to have a very close API in @fast-check/jest.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.