Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[core] Report performance test results on each PR #3551

Merged
merged 7 commits into from Jan 14, 2022

Conversation

m4theushw
Copy link
Member

@m4theushw m4theushw commented Jan 5, 2022

This PR removes the performance tests that were running as unit test. Instead, they run now in a different environment, with the React production build and closer to how the user interacts with the grid. The results will be reported as a comment in each PR, that means that the PR, initially, won't fail if a regression is introduced.

The comparison between these results and a "gold master" (assumed to be the last released version) will be done after the merge and once we have a performance snapshot for a release.

As inspiration: https://github.com/saucelabs/performance-CI-demo/blob/main/README.md

@m4theushw

This comment has been minimized.

@m4theushw m4theushw added the core Infrastructure work going on behind the scenes label Jan 5, 2022
@github-actions
Copy link

github-actions bot commented Jan 7, 2022

This pull request has conflicts, please resolve those before we can evaluate the pull request.

@github-actions github-actions bot added the PR: out-of-date The pull request has merge conflicts and can't be merged label Jan 7, 2022
@oliviertassinari
Copy link
Member

oliviertassinari commented Jan 11, 2022

Could you set DANGER_GITHUB_API_TOKEN on this repo?

@m4theushw Done, added to CircleCI. The comment should come from https://github.com/mui-bot going forward. I have followed https://danger.systems/guides/getting_started.html

@github-actions github-actions bot removed the PR: out-of-date The pull request has merge conflicts and can't be merged label Jan 11, 2022
@mui-bot
Copy link

mui-bot commented Jan 11, 2022

These are the results for the performance tests:

Test case Unit Min Max Median Mean σ
Filter 100k rows ms 239.7 375.2 311.6 303 46.9
Sort 100k rows ms 472.2 794.8 718 672.74 125.891
Select 100k rows ms 217.3 315.5 268 267.28 32.912
Deselect 100k rows ms 99.7 277.9 231.1 204.3 58.898

Generated by 🚫 dangerJS against c44a33d

@m4theushw m4theushw marked this pull request as ready for review January 12, 2022 14:37
@m4theushw m4theushw requested a review from a team January 12, 2022 15:01
@flaviendelangle
Copy link
Member

@m4theushw could we add the standard deviation or the variance to have a glance of the value accurateness ?
I will not treat the same way a variation à 10ms if the variance is of 1ms or 50ms.

@m4theushw
Copy link
Member Author

m4theushw commented Jan 12, 2022

@flaviendelangle I added the standard deviation. As you can see, the values vary substantially, probably because integration between Playwright and the Chrome DevTools protocol is slow. This means that we won't be able to see if a PR has caused a small impact on performance. However, it's going to be useful for situations where a change causes a drastic performance hit, in which a unit test, using a small dataset, wouldn't fail due timeout. In 154cc7e I intentionally caused a performance degradation (ignore the failing unit tests). The average time to select all rows is 60x higher. This is a clear signal that something is wrong.

I'll revert 154cc7e once you react to this comment.

@flaviendelangle
Copy link
Member

Seems perfect 👍

@m4theushw m4theushw self-assigned this Jan 14, 2022
@m4theushw m4theushw merged commit dd8c833 into mui:master Jan 14, 2022
@m4theushw m4theushw deleted the performance-tests branch January 14, 2022 02:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core Infrastructure work going on behind the scenes performance
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants