Test retries #6498

palmerj3 · 2018-06-19T23:14:16Z

Summary

This PR introduces a new feature to Jest that allows you to re-run failed tests automatically n-times.
This is configurable via the Jest object.

Example:

jest.retryTimes(3);
By default Jest will not re-run any failed tests.

The rationale behind this feature is that many teams use Jest for Integration and E2E tests. Higher level tests are much more likely to be flaky, and flakiness is often not the fault of the test itself (e.g. network problems).

This option provides the user a much better experience in CI since tests can be retried a few times before failing. The current behavior requires entire CI jobs to be re-triggered when a single test fails.

This PR also introduces a new field in TestResults for reporters to make use of. It is called "invocations" and represents the number of times a given test was executed.

This functionality is only available if you use jest-circus!

Test plan

Added a e2e test to ensure that tests are only re-run when jest.retryTimes is set and jest-circus is used.

codecov-io · 2018-06-19T23:30:08Z

Codecov Report

Merging #6498 into master will decrease coverage by 0.08%.
The diff coverage is 7.14%.

@@            Coverage Diff             @@
##           master    #6498      +/-   ##
==========================================
- Coverage   63.83%   63.75%   -0.09%     
==========================================
  Files         234      234              
  Lines        8797     8810      +13     
  Branches        3        3              
==========================================
+ Hits         5616     5617       +1     
- Misses       3180     3192      +12     
  Partials        1        1

Impacted Files	Coverage Δ
packages/jest-circus/src/utils.js	`21.66% <ø> (ø)`	⬆️
.../src/legacy_code_todo_rewrite/jest_adapter_init.js	`0% <ø> (ø)`	⬆️
packages/jest-circus/src/run.js	`0% <0%> (ø)`	⬆️
packages/jest-circus/src/event_handler.js	`12.67% <0%> (-0.19%)`	⬇️
packages/jest-runtime/src/index.js	`72.53% <25%> (-0.37%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5010a5c...19bf8a6. Read the comment docs.

SimenB · 2018-06-20T05:42:20Z

packages/jest-cli/src/cli/args.js

@@ -51,6 +51,17 @@ export const check = (argv: Argv) => {
    );
  }

+  const intRetries = parseInt(argv.testRetries, 10);
+  if (
+    (argv.hasOwnProperty('testRetries') && isNaN(intRetries)) ||


Number.isNaN

SimenB

I like this idea! Making Jest more useful for integration tests would be awesome.

This is missing tests, docs and a changelog entry, but the code itself LGTM 🙂

thymikee · 2018-06-20T05:48:51Z

packages/jest-circus/src/run.js

@@ -28,30 +28,41 @@ import {

 const Promise = getOriginalPromise();

-const run = async (): Promise<RunResult> => {
+const run = async (numRetries: number): Promise<RunResult> => {


Can we stay consistent with the naming and use testRetries instead?

palmerj3 · 2018-06-20T10:17:21Z

Great! I'll make these changes today and update the PR.

palmerj3 · 2018-06-20T10:28:42Z

Also I have code working locally which updates the reporter API adding a new field for TestResults called invocations. I'm not sure how useful it would be so I haven't pushed it up. But I like the idea of a reporter being able to track which tests are retried since it's a signal about potential flakiness. Let me know your thoughts on that. I'm happy to include that if you think it's worthwhile.

SimenB · 2018-06-20T15:00:14Z

Yeah seems useful. Can use it to inform about flakiness over time

aaronabramov · 2018-06-20T18:32:39Z

i was also thinking about test.retries(5, 'does stuff', () => {}); api to retry some nasty tests explicitly.
also it might make sense to reschedule these tests to the end of the file after other tests complete. e.g.:

✅test 1
❌test 2
✅test 3
✅test 4
---
✅test 2 (tetry)

the reason is that sometimes when you run e2e integration tests there's can be a spike in CPU usage (another thread decided to spawn something?) and the test times out. If you retry it right away the spike might be still going on and it'll fail again, but if you first try to execute everything else, see what passes, and then re-execute tests that failed you might have enough time for the CPU spike to die down.

aaronabramov · 2018-06-20T18:34:41Z

also.. i noticed when executing fat integration tests, sometimes they timeout, jest moves on to the next test and the previous test eventually throws something. Jest treats it as a global error (because it does not know where it came from) and fails whatever current test is running, which results in a pretty confusing behavior.
With retries it will probably start throwing even more errors

palmerj3 · 2018-06-20T20:35:47Z

@aaronabramov awesome! Is test.retry something you want me to implement as part of this or entirely to replace the CLI/Config-based approach?

Also regarding errors thrown after tests - not really sure the best way to make tests run later on. Push to the end of the test array? How would I do that?

aaronabramov · 2018-06-20T21:00:21Z

@palmerj3 those we just my observations from last time i tried it! we can definitely iterate on those ideas later!
i don't mind CLI, but it kind of feels like it's too many config options (plus it won't be relevant for other runners)
maybe we should start by implementing it on the jest object? similar to jest.setTimeout()?
like

jest.retryTimes(3);

or something like that

this way we can still set it globally from a setupFile

palmerj3 · 2018-06-21T01:11:18Z

Awesome - you've given me a different perspective for approaching this problem. I'm going to try a few things out (jest obj, test.retry, etc) and see what comes of it. Thanks!

palmerj3 · 2018-06-21T01:55:58Z

@aaronabramov I guess the thing I'm confused about is how I would responsibly go about retrying tests via the jest object? I can easily set a global to know how many retries are required in a given context but the act of actually retrying would have to be implemented by the test runner unless I override the test/it global and orchestrate from jest-runtime.

aaronabramov · 2018-06-21T02:39:36Z

@palmerj3 yeah, i was thinking about exactly how jest.setTimeout() does it.
it sets the global value (that defaults to 1), that lives in jest-circus state and then once we're running tests, we access this value and check how many retries we have

palmerj3 · 2018-06-21T11:03:55Z

Ok cool that's how I'll do it

palmerj3 · 2018-06-21T17:29:46Z

@aaronabramov @SimenB if you get a chance please review again.

CI seems to be failing and I'm having a hard time debugging that... tests all pass locally on node 9 and node 10 so I'm not understanding why things fail on CI.

That said: I updated the PR so test retries are configured via the Jest object and I updated the reporter API so "invocations" field is available in TestResult.

Take a look when you can and offer ideas on how to fix CI :)

SimenB · 2018-06-21T19:34:32Z

packages/jest-runtime/src/index.js

@@ -830,6 +830,11 @@ class Runtime {
      return jestObject;
    };

+    const retryTimes = (numTestRetries: number) => {
+      this._environment.global.retryTimes = numTestRetries;


can we put it on the jest object instead of as a global?

I'll see if that will work

@SimenB so I don't think that's going to work. From the context of jest-circus/run.js I don't have access to the jest object. Even if I did I think I would have to add both a getter and a setter for that to work. What do you think?

a clean (and private) way of speaking across that boundary would be awesome, but no need to tackle it at this moment 🙂

last time i did it with a Symbol that is attached to global. which is pretty much the same thing, but you only know it's there if you know what you're looking for :)

could we do that? Polluting the global namespace in user code seems icky

EDIT: "that" being the symbol Aaron suggested, since this comment was not kept in context...

SimenB · 2018-06-21T19:37:36Z

e2e/__tests__/test_retries.test.js

+    runJest(
+      'test-retries',
+      ['--config', JSON.stringify(reporterConfig), 'retry.test.js'],
+      {useJestCircus: true},


you can do skipOnJasmine instead, then it will run just with circus

https://github.com/facebook/jest/blob/d1ce3cd0135b5a8b64f84998fd5f36be4afb2bef/scripts/ConditionalTest.js#L17

Ah perfect thanks!

SimenB · 2018-06-21T19:39:49Z

docs/JestObjectAPI.md

@@ -312,6 +313,21 @@ test('works too', () => {

 Returns the `jest` object for chaining.

+### `jest.retryTimes()`


should mention it just works with circus

I mention in that section that it works with jest-circus. Should I be more explicit?

Where is it mentioned?

Ah my mistake sorry. I mentioned it in the changelog not the docs. Will update.

SimenB · 2018-06-21T20:41:34Z

e2e/runJest.js

@@ -21,6 +21,7 @@ const JEST_PATH = path.resolve(__dirname, '../packages/jest-cli/bin/jest.js');
 type RunJestOptions = {
  nodePath?: string,
  skipPkgJsonCheck?: boolean, // don't complain if can't find package.json
+  useJestCircus?: boolean,


we don't need the changes in this file now, right?

palmerj3 · 2018-06-22T01:39:46Z

@aaronabramov @SimenB I think the PR is ready. I've addressed all the feedback so far.

But I'm getting a weird (unrelated?) error on some CI checks that I can't make sense of. All things pass locally on node 9 and node 10.

TypeError: Cannot read property 'prototype' of undefined

Something related to graceful-fs

SimenB · 2018-06-22T15:37:10Z

e2e/__tests__/test_retries.test.js

+      ],
+    };
+
+    // Test retries only available via JEST_CIRCUS


outdated comment, right?

Yep. Pushing a commit now to fix.

All set - still getting CI errors due to the graceful-fs dependency bump yesterday (I believe)

The comment is still there :D

CI failures are not because of graceful-fs bump (there wasn't any afaik), but started to happen since Node 10.5

SimenB · 2018-06-22T15:37:41Z

e2e/runJest.js

@@ -49,12 +49,15 @@ function runJest(
    );
  }

-  const env = options.nodePath
-    ? Object.assign({}, process.env, {
+  const nodePathOverrides = options.nodePath


needed change?

I can revert that back to the original

palmerj3 · 2018-06-22T18:51:26Z

I really need to learn how to software :) Outdated comments removed.

aaronabramov · 2018-06-22T18:48:47Z

e2e/snapshot-escape/__tests__/snapshot.test.js

@@ -8,3 +8,4 @@

 // prettier-ignore
 test('escape strings', () => expect('one: \\\'').toMatchSnapshot());
+test('escape strings two', () => expect('two: \'"').toMatchSnapshot());


i think this line is not supposed to be committed.
i believe it's coming from our old janky snapshot tests that modifies our fixtures (that are under git and not in /tmp) and if you interrupt the test before it's done this thing will stay there :( (hence all other modifications to snapshots you had to make)

check e2e/__tests__/snapshot.test.js

Ah ok - should I remove this file or that line?

I saw other PRs failing with the same error

aaronabramov · 2018-06-22T18:53:17Z

packages/jest-runtime/src/index.js

@@ -830,6 +830,11 @@ class Runtime {
      return jestObject;
    };

+    const retryTimes = (numTestRetries: number) => {
+      this._environment.global.retryTimes = numTestRetries;


last time i did it with a Symbol that is attached to global. which is pretty much the same thing, but you only know it's there if you know what you're looking for :)

palmerj3 · 2018-06-22T22:29:57Z

@aaronabramov so I removed that one test case but still getting the same error. Happy to revert that commit if you like. But if you have other ideas for things I can check I can do that too.

palmerj3 · 2018-06-25T14:34:53Z

@SimenB all set

palmerj3 · 2018-06-26T12:35:12Z

Merge? :)

SimenB · 2018-06-26T18:19:33Z

FB employees typically merge the larger changes :)

palmerj3 · 2018-06-26T20:36:51Z

@aaronabramov any more thoughts on this? No rush on my end - would just rather not have to rebase again :)

aaronabramov · 2018-06-26T21:28:35Z

it looks good to me!

This Pull Request updates dependency [jest](https://github.com/facebook/jest) from `v23.1.0` to `v23.2.0` <details> <summary>Release Notes</summary> ### [`v23.2.0`](https://github.com/facebook/jest/blob/master/CHANGELOG.md#2320) [Compare Source](jestjs/jest@v23.1.0...v23.2.0) ##### Features - `[jest-each]` Add support for keyPaths in test titles ([#6457](`jestjs/jest#6457)) - `[jest-cli]` Add `jest --init` option that generates a basic configuration file with a short description for each option ([#6442](`jestjs/jest#6442)) - `[jest.retryTimes]` Add `jest.retryTimes()` option that allows failed tests to be retried n-times when using jest-circus. ([#6498](`jestjs/jest#6498)) ##### Fixes - `[jest-cli]` Add check to make sure one or more tests have run before notifying when using `--notify` ([#6495](`jestjs/jest#6495)) - `[jest-cli]` Pass `globalConfig` as a parameter to `globalSetup` and `globalTeardown` functions ([#6486](`jestjs/jest#6486)) - `[jest-config]` Add missing options to the `defaults` object ([#6428](`jestjs/jest#6428)) - `[expect]` Using symbolic property names in arrays no longer causes the `toEqual` matcher to fail ([#6391](`jestjs/jest#6391)) - `[expect]` `toEqual` no longer tries to compare non-enumerable symbolic properties, to be consistent with non-symbolic properties. ([#6398](`jestjs/jest#6398)) - `[jest-util]` `console.timeEnd` now properly log elapsed time in milliseconds. ([#6456](`jestjs/jest#6456)) - `[jest-mock]` Fix `MockNativeMethods` access in react-native `jest.mock()` ([#6505](`jestjs/jest#6505)) ##### Chore & Maintenance - `[docs]` Add jest-each docs for 1 dimensional arrays ([#6444](`jestjs/jest#6444)) --- </details> --- This PR has been generated by [Renovate Bot](https://renovatebot.com).

github-actions · 2021-05-12T12:07:28Z

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.

facebook-github-bot added the cla signed label Jun 19, 2018

palmerj3 force-pushed the testRetries branch from 226e60f to fb1422e Compare June 20, 2018 03:46

SimenB requested a review from aaronabramov June 20, 2018 05:41

SimenB reviewed Jun 20, 2018

View reviewed changes

SimenB requested changes Jun 20, 2018

View reviewed changes

thymikee reviewed Jun 20, 2018

View reviewed changes

palmerj3 force-pushed the testRetries branch from fb1422e to e011fe9 Compare June 21, 2018 01:26

SimenB reviewed Jun 21, 2018

View reviewed changes

palmerj3 force-pushed the testRetries branch from ba295db to 9e027c2 Compare June 22, 2018 01:05

SimenB approved these changes Jun 22, 2018

View reviewed changes

aaronabramov approved these changes Jun 22, 2018

View reviewed changes

palmerj3 added 5 commits June 25, 2018 09:27

Remove useJestCircus boolean on runTest

09fb974

Remove outdated comment. Revert runJest environment override logic.

b346cc2

Removed outdated comment from test_retries test

b9d65f8

Update snapshot tests

20667aa

Update Jest Object docs for retryTimes. Use symbol for global lookup.

19bf8a6

palmerj3 force-pushed the testRetries branch from 6e4c1aa to 19bf8a6 Compare June 25, 2018 13:27

SimenB approved these changes Jun 25, 2018

View reviewed changes

aaronabramov merged commit 812dc12 into jestjs:master Jun 26, 2018

palmerj3 deleted the testRetries branch June 26, 2018 22:12

dependencies bot mentioned this pull request Nov 1, 2018

Update jest in / from 22.1.4 to 23.6.0 leonardofaria/leozera-ui#250

Closed

snyk-bot mentioned this pull request Mar 17, 2020

[Snyk] Upgrade expect from 23.0.0 to 23.6.0 yowainwright/react-workshop#8

Merged

github-actions bot locked as resolved and limited conversation to collaborators May 12, 2021

		@@ -312,6 +313,21 @@ test('works too', () => {

		Returns the `jest` object for chaining.

		### `jest.retryTimes()`

Test retries #6498

Test retries #6498

Conversation

palmerj3 commented Jun 19, 2018 • edited

Summary

Test plan

codecov-io commented Jun 19, 2018 • edited

Codecov Report

Choose a reason for hiding this comment

SimenB left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

palmerj3 commented Jun 20, 2018

palmerj3 commented Jun 20, 2018

SimenB commented Jun 20, 2018

aaronabramov commented Jun 20, 2018

aaronabramov commented Jun 20, 2018

palmerj3 commented Jun 20, 2018

aaronabramov commented Jun 20, 2018 • edited

palmerj3 commented Jun 21, 2018

palmerj3 commented Jun 21, 2018

aaronabramov commented Jun 21, 2018

palmerj3 commented Jun 21, 2018

palmerj3 commented Jun 21, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SimenB Jun 25, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

palmerj3 commented Jun 22, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

palmerj3 commented Jun 22, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

palmerj3 commented Jun 22, 2018

palmerj3 commented Jun 25, 2018

palmerj3 commented Jun 26, 2018

SimenB commented Jun 26, 2018

palmerj3 commented Jun 26, 2018

aaronabramov commented Jun 26, 2018

github-actions bot commented May 12, 2021

palmerj3 commented Jun 19, 2018 •

edited

codecov-io commented Jun 19, 2018 •

edited

SimenB left a comment •

edited

aaronabramov commented Jun 20, 2018 •

edited

SimenB Jun 25, 2018 •

edited