Add update-backlog subcommand
#83
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ErichDonGubler
commented
Apr 8, 2024
ErichDonGubler
force-pushed
the
update-backlog
branch
3 times, most recently
from
ef6b6ee
to
a856ba3
Compare
ErichDonGubler
force-pushed
the
update-backlog
branch
3 times, most recently
from
9fc58ea
to
c91b937
Compare
ErichDonGubler
force-pushed
the
update-backlog
branch
from
c91b937
to
16eea71
Compare
ErichDonGubler
commented
May 30, 2024
| }; | ||
| cases[(platform, build_profile)] = case; | ||
| } | ||
| if !subtests.is_empty() { |
There was a problem hiding this comment.
issue: Got a bug here where expected test outcomes are ignored when subtests are present. We should take both into account.
TODO: Validate that this is useful.
ErichDonGubler
force-pushed
the
update-backlog
branch
from
16eea71
to
bbca91c
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We at Mozilla recently added the
implementation-status: backlogproperty to all test cases (see upstream docs on metadata), so we can create a workflow for incrementally making WebGPU CTS visible to Firefox's CI sheriffing team. The idea is to incrementally remove it as we determine contradictory test results are worth filing bugs for. We now useimplementation-statusfor both tier 2 and tier 3 of Firefox CI. I'm currently calling the migration of a test from a less stable tier to a more stable tier by removingimplementation-status: backloga "promotion".We do experiments in promoting tests according to heuristics like "promote permanently
PASSing tests" (already implemented here), "promote tests that aren't observed toFAILorCRASH" (to be implemented in #109), with more to come. The workflow then becomes:wptreport.jsonfiles.update-expectedas desired forbacklogged tests, and create a commit based on changes.update-backlogto tentatively promote tests. Commit and submit to CI as an experiment.We consider using
implementation-status: backlogto be valuable because:wptrunnerhas CLI support for filtering tests by implementation status, so the lift to adjust our CI was light.expected; sometimes, we want to model that a test should pass, but not block CI on it yet. This came up recently with bug 1897131.Original OP
I'm currently planning on separating the corpus of WebGPU CTS test runs in Firefox CI into testing based on the
implementation-status(see bug 1873687). I'm using this PR as a way to explore automatically changing theimplementation-status.