Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test(queryClient): Remove flaky perfomance tests #6946

Merged
merged 1 commit into from
Feb 24, 2024
Merged

test(queryClient): Remove flaky perfomance tests #6946

merged 1 commit into from
Feb 24, 2024

Conversation

SebKranz
Copy link
Contributor

@SebKranz SebKranz commented Feb 20, 2024 •
edited

While investigating flaky performance tests added in #6918, I found that other unrelated tests would randomly fail.
This happened mainly to tests that relied on timers firing at exact points, such as useIsMutating > should return the number of fetching mutations:

 FAIL  |@tanstack/react-query| src/__tests__/useMutationState.test.tsx > useIsMutating > should return the number of fetching mutations
AssertionError: expected [ +0, 1, 2, +0 ] to deeply equal [ +0, 1, 2, 1, +0 ]

Ignored nodes: comments, script, style
<html>
  <head />
  <body>
    <div>
      <div />
    </div>
  </body>
</html>

- Expected
+ Received

  Array [
    0,
    1,
    2,
-   1,
    0,
  ]

 ❯ src/__tests__/useMutationState.test.tsx:62:49

My guess here is that the event loop is delaying the timers to such a degree that they fired back-to-back and caused React to batch the re-render.

Possible solutions for this would be to...

  • ...occasionally defer to the Task Queue while running the performance test.
  • ...simplify the performance test to run in less time. This would lead to flakiness due to variance in cpu load.
  • ...refactor all other tests to correctly handle delayed timer responses.
  • ...run these performance tests non-concurrently, after all other tests. This would probably require a larger reorganisation of Tests since NX is running these suites concurrently, not Vitest.

All of these solutions are either impractical or disproportional to the value of the performance tests. Therefore, I think it's best to simply remove the performance tests.

@vercel Vercel
Copy link

vercel bot commented Feb 20, 2024 •
edited

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Comments Updated (UTC)
query ⬜️ Ignored (Inspect) Visit Preview Feb 20, 2024 1:06pm

@nx-cloud Nx Cloud
Copy link

nx-cloud bot commented Feb 20, 2024 •
edited

☁️ Nx Cloud Report

CI is running/has finished running commands for commit 518cb2c. As they complete they will appear below. Click to see the status, the terminal output, and the build insights.

📂 See all runs for this CI Pipeline Execution

✅ Successfully ran 1 target

Sent with 💌 from NxCloud.

@codesandbox-ci CodeSandbox CI
Copy link

This pull request is automatically built and testable in CodeSandbox.

To see build info of the built libraries, click here or the icon next to each commit SHA.

Latest deployment of this branch, based on commit 518cb2c:

Sandbox Source
@tanstack/query-example-angular-basic Configuration
@tanstack/query-example-react-basic-typescript Configuration
@tanstack/query-example-solid-basic-typescript Configuration
@tanstack/query-example-svelte-basic Configuration
@tanstack/query-example-vue-basic Configuration

@SebKranz SebKranz mentioned this pull request Feb 20, 2024
Closed
@TkDodo TkDodo merged commit 747aa78 into TanStack:main Feb 24, 2024
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@SebKranz @TkDodo