feat(test runner): Use timing file to balance shards #30388
Draft
+110
−3
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This feature came from this issue: #17969
The idea is to use the JSON reporter and extract the duration of each test, then map the test groups based on the duration so the shards are all as equal as possible, the amount of tests per shard is not relevant here, only the duration of the tests.
If there are tests that are not recorded in the JSON report they will be distributed like they currently do based on the amount.
This is still in a POC state to get some feedback, but the functionality works well.
Some open questions:
At the end of the day unless you come up with some cloud solution, the 'infra' for this feature would need to be done mostly manually, a static JSON report is optional but might get irrelevant very fast.
On my project I've managed it by averaging the durations of the last 10 successful runs, which is being updated to a CDN from CI.
Some more details on current POC can be seen here:
#17969 (comment)
#17969 (comment)