Add --fail-skip/todo opts, better skip reporting #901
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
isaacs
force-pushed
the
isaacs/fail-skip-todo
branch
from
f0d1452
to
62efd4f
Compare
This reports 'no tests found' as the skip reason for any test that does not have a parent (so, typically, just the root TAP test) and also emits no tests with a 1..0 plan, even if it lacks a skipReason in the plan. Also, if `failSkip` is set in the test options (defaulting to `env.TAP_FAIL_SKIP` in the root test) then any skipped test points will be treated as failures. Ditto todo tests with `failTodo`. This makes it possible to run `tap --fail-skip --fail-todo -s todo.txt` to easily work through a backlog of skipped tests. Doesn't exactly play nicely with --grep and --only, but it's hard to know what to do in that case. You're telling tap that you want to fail if any tests are skipped, and also telling it to skip tests. It seems reasonable to assume that you don't want it to fail for tests skipped with --grep or --only, since it should be as if those tests just didn't *exist*, so probably going to need to figure out another way to signal this. Fix: #892
isaacs
force-pushed
the
isaacs/fail-skip-todo
branch
from
62efd4f
to
4dea127
Compare
This makes it possible to use `--grep` and `--only` along with `--fail-skip`, without making a ton of excessive noise. If you *want* `--fail-skip` to fail on filtered tests, then you can set `--no-filter-quietly` to force the skip messages. Also, it's kind of nicer when filtering tests anyway, because it provides a way to not have them show a bunch of noisy repetitive skip messages.
isaacs
force-pushed
the
isaacs/fail-skip-todo
branch
from
4dea127
to
d63baad
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This reports 'no tests found' as the skip reason for any test that does
not have a parent (so, typically, just the root TAP test) and also emits
no tests with a 1..0 plan, even if it lacks a skipReason in the plan.
Also, if
failSkipis set in the test options (defaulting toenv.TAP_FAIL_SKIPin the root test) then any skipped test points willbe treated as failures. Ditto todo tests with
failTodo.This makes it possible to run
tap --fail-skip --fail-todo -s todo.txtto easily work through a backlog of skipped tests.
Second commit adds
--filter-quietly, which defaults to true when--fail-skipis set, making it possible to use--fail-skipalong with--grepand--onlywithout a ton of noise.Fix: #892