Add new SKIP macro for skipping tests at runtime (#2360)

* Add new SKIP macro for skipping tests at runtime

This adds a new `SKIP` macro for dynamically skipping tests at runtime.
The "skipped" status of a test case is treated as a first-class citizen,
like "succeeded" or "failed", and is reported with a new color on the
console.

* Don't show "skipped assertions" in console/compact reporters

Also extend skip tests to cover a few more use cases.

* Return exit code 4 if all test cases are skipped

* Use LightGrey for the skip colour

This isn't great, but is better than the deep blue that was borderline
invisible on dark backgrounds. The fix is to redo the colouring
a bit, including introducing light-blue that is actually visible.

* Add support for explicit skips in all reporters

* --allow-running-no-tests also allows all tests to be skipped

* Add docs for SKIP macro, deprecate IEventListener::skipTest

Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>

docs/Readme.md

@@ -11,6 +11,7 @@ Once you're up and running consider the following reference material.
 * [Logging macros](logging.md#top)
 * [Test cases and sections](test-cases-and-sections.md#top)
 * [Test fixtures](test-fixtures.md#top)
+* [Skipping tests at runtime](skipping.md#top)
 * [Reporters (output customization)](reporters.md#top)
 * [Event Listeners](event-listeners.md#top)
 * [Data Generators (value parameterized tests)](generators.md#top)

docs/command-line.md

@@ -561,10 +561,10 @@ processes, as is done with the [Bazel test sharding](https://docs.bazel.build/ve
 
 > Introduced in Catch2 3.0.1.
 
-By default, Catch2 test binaries return non-0 exit code if no tests were
-run, e.g. if the binary was compiled with no tests, or the provided test
-spec matched no tests. This flag overrides that, so a test run with no
-tests still returns 0.
+By default, Catch2 test binaries return non-0 exit code if no tests were run,
+e.g. if the binary was compiled with no tests, the provided test spec matched no
+tests, or all tests [were skipped at runtime](skipping.md#top). This flag
+overrides that, so a test run with no tests still returns 0.
 
 ## Output verbosity
 ```

docs/deprecations.md

@@ -26,6 +26,15 @@ to accurately probe the environment for this information so the flag
 where it will export `BAZEL_TEST=1` for purposes like the above. Catch2
 will now instead inspect the environment instead of relying on build configuration.
 
+### `IEventLister::skipTest( TestCaseInfo const& testInfo )`
+
+This event (including implementations in derived classes such as `ReporterBase`)
+is deprecated and will be removed in the next major release. It is currently
+invoked for all test cases that are not going to be executed due to the test run
+being aborted (when using `--abort` or `--abortx`). It is however
+**NOT** invoked for test cases that are [explicitly skipped using the `SKIP`
+macro](skipping.md#top).
+
 ---
 
 [Home](Readme.md#top)

docs/skipping.md

@@ -0,0 +1,72 @@
+<a id="top"></a>
+# Skipping Test Cases at Runtime
+
+> [Introduced](https://github.com/catchorg/Catch2/pull/2360) in Catch2 X.Y.Z.
+
+In some situations it may not be possible to meaningfully execute a test case, for example when the system under test is missing certain hardware capabilities.
+If the required conditions can only be determined at runtime, it often doesn't make sense to consider such a test case as either passed or failed, because it simply can not run at all.
+To properly express such scenarios, Catch2 allows to explicitly _skip_ test cases, using the `SKIP` macro:
+
+**SKIP(** _message expression_ **)**
+
+Example usage:
+
+```c++
+TEST_CASE("copy files between drives") {
+    if(getNumberOfHardDrives() < 2) {
+        SKIP("at least two hard drives required");
+    }
+    // ...
+}
+```
+
+This test case is then reported as _skipped_ instead of _passed_ or _failed_.
+
+The `SKIP` macro behaves similarly to an explicit [`FAIL`](logging.md#top), in that it is the last expression that will be executed:
+
+```c++
+TEST_CASE("my test") {
+    printf("foo");
+    SKIP();
+    printf("bar"); // not printed
+}
+```
+
+However a failed assertion _before_ a `SKIP` still causes the entire test case to fail:
+
+```c++
+TEST_CASE("failing test") {
+    CHECK(1 == 2);
+    SKIP();
+}
+```
+
+## Interaction with Sections and Generators
+
+Sections, nested sections as well as test cases with [generators](generators.md#top) can all be individually skipped, with the rest executing as usual:
+
+```c++
+TEST_CASE("complex test case") {
+  int value = GENERATE(2, 4, 6);
+  SECTION("a") {
+    SECTION("a1") { CHECK(value < 8); }
+    SECTION("a2") {
+      if (value == 4) {
+        SKIP();
+      }
+      CHECK(value % 2 == 0);
+    }
+  }
+}
+```
+
+This test case will report 5 passing assertions; one for each of the three values in section `a1`, as well as one for each in `a2`, except for when `value == 4`.
+
+Note that as soon as one section is skipped, the entire test case will be reported as _skipped_ (unless there is a failing assertion, in which case it will be reported as _failed_ instead).
+
+If all test cases in a run are skipped, Catch2 returns a non-zero exit code by default.
+This can be overridden using the [--allow-running-no-tests](command-line.md#no-tests-override) flag.
+
+---
+
+[Home](Readme.md#top)

src/catch2/catch_session.cpp

@@ -341,6 +341,12 @@ namespace Catch {
                 return 2;
             }
 
+            if ( totals.testCases.total() > 0 &&
+                 totals.testCases.total() == totals.testCases.skipped
+                && !m_config->zeroTestsCountAsSuccess() ) {
+                return 4;
+            }
+
             // Note that on unices only the lower 8 bits are usually used, clamping
             // the return value to 255 prevents false negative when some multiple
             // of 256 tests has failed

src/catch2/catch_test_macros.hpp

@@ -49,6 +49,7 @@
   #define CATCH_FAIL( ... ) INTERNAL_CATCH_MSG( "CATCH_FAIL", Catch::ResultWas::ExplicitFailure, Catch::ResultDisposition::Normal, __VA_ARGS__ )
   #define CATCH_FAIL_CHECK( ... ) INTERNAL_CATCH_MSG( "CATCH_FAIL_CHECK", Catch::ResultWas::ExplicitFailure, Catch::ResultDisposition::ContinueOnFailure, __VA_ARGS__ )
   #define CATCH_SUCCEED( ... ) INTERNAL_CATCH_MSG( "CATCH_SUCCEED", Catch::ResultWas::Ok, Catch::ResultDisposition::ContinueOnFailure, __VA_ARGS__ )
+  #define CATCH_SKIP( ... ) INTERNAL_CATCH_MSG( "SKIP", Catch::ResultWas::ExplicitSkip, Catch::ResultDisposition::Normal, __VA_ARGS__ )
 
 
   #if !defined(CATCH_CONFIG_RUNTIME_STATIC_REQUIRE)
@@ -102,6 +103,7 @@
   #define CATCH_FAIL( ... ) (void)(0)
   #define CATCH_FAIL_CHECK( ... ) (void)(0)
   #define CATCH_SUCCEED( ... ) (void)(0)
+  #define CATCH_SKIP( ... ) (void)(0)
 
   #define CATCH_STATIC_REQUIRE( ... )       (void)(0)
   #define CATCH_STATIC_REQUIRE_FALSE( ... ) (void)(0)
@@ -146,6 +148,7 @@
   #define FAIL( ... ) INTERNAL_CATCH_MSG( "FAIL", Catch::ResultWas::ExplicitFailure, Catch::ResultDisposition::Normal, __VA_ARGS__ )
   #define FAIL_CHECK( ... ) INTERNAL_CATCH_MSG( "FAIL_CHECK", Catch::ResultWas::ExplicitFailure, Catch::ResultDisposition::ContinueOnFailure, __VA_ARGS__ )
   #define SUCCEED( ... ) INTERNAL_CATCH_MSG( "SUCCEED", Catch::ResultWas::Ok, Catch::ResultDisposition::ContinueOnFailure, __VA_ARGS__ )
+  #define SKIP( ... ) INTERNAL_CATCH_MSG( "SKIP", Catch::ResultWas::ExplicitSkip, Catch::ResultDisposition::Normal, __VA_ARGS__ )
 
 
   #if !defined(CATCH_CONFIG_RUNTIME_STATIC_REQUIRE)
@@ -198,6 +201,7 @@
   #define FAIL( ... ) (void)(0)
   #define FAIL_CHECK( ... ) (void)(0)
   #define SUCCEED( ... ) (void)(0)
+  #define SKIP( ... ) (void)(0)
 
   #define STATIC_REQUIRE( ... )       (void)(0)
   #define STATIC_REQUIRE_FALSE( ... ) (void)(0)

src/catch2/catch_totals.cpp

@@ -14,21 +14,23 @@ namespace Catch {
         diff.passed = passed - other.passed;
         diff.failed = failed - other.failed;
         diff.failedButOk = failedButOk - other.failedButOk;
+        diff.skipped = skipped - other.skipped;
         return diff;
     }
 
     Counts& Counts::operator += ( Counts const& other ) {
         passed += other.passed;
         failed += other.failed;
         failedButOk += other.failedButOk;
+        skipped += other.skipped;
         return *this;
     }
 
     std::uint64_t Counts::total() const {
-        return passed + failed + failedButOk;
+        return passed + failed + failedButOk + skipped;
     }
     bool Counts::allPassed() const {
-        return failed == 0 && failedButOk == 0;
+        return failed == 0 && failedButOk == 0 && skipped == 0;
     }
     bool Counts::allOk() const {
         return failed == 0;
@@ -53,6 +55,8 @@ namespace Catch {
             ++diff.testCases.failed;
         else if( diff.assertions.failedButOk > 0 )
             ++diff.testCases.failedButOk;
+        else if ( diff.assertions.skipped > 0 )
+            ++ diff.testCases.skipped;
         else
             ++diff.testCases.passed;
         return diff;

src/catch2/catch_totals.hpp

@@ -23,6 +23,7 @@ namespace Catch {
         std::uint64_t passed = 0;
         std::uint64_t failed = 0;
         std::uint64_t failedButOk = 0;
+        std::uint64_t skipped = 0;
     };
 
     struct Totals {

src/catch2/interfaces/catch_interfaces_reporter.hpp

@@ -242,7 +242,12 @@ namespace Catch {
          */
         virtual void testRunEnded( TestRunStats const& testRunStats ) = 0;
 
-        //! Called with test cases that are skipped due to the test run aborting
+        /**
+         * Called with test cases that are skipped due to the test run aborting.
+         * NOT called for test cases that are explicitly skipped using the `SKIP` macro.
+         *
+         * Deprecated - will be removed in the next major release.
+         */
         virtual void skipTest( TestCaseInfo const& testInfo ) = 0;
 
         //! Called if a fatal error (signal/structured exception) occured

src/catch2/internal/catch_assertion_handler.cpp

@@ -50,6 +50,13 @@ namespace Catch {
         if (m_reaction.shouldThrow) {
             throw_test_failure_exception();
         }
+        if ( m_reaction.shouldSkip ) {
+#if !defined( CATCH_CONFIG_DISABLE_EXCEPTIONS )
+            throw Catch::TestSkipException();
+#else
+            CATCH_ERROR( "Explicitly skipping tests during runtime requires exceptions" );
+#endif
+        }
     }
     void AssertionHandler::setCompleted() {
         m_completed = true;

src/catch2/internal/catch_assertion_handler.hpp

@@ -22,6 +22,7 @@ namespace Catch {
     struct AssertionReaction {
         bool shouldDebugBreak = false;
         bool shouldThrow = false;
+        bool shouldSkip = false;
     };
 
     class AssertionHandler {

src/catch2/internal/catch_console_colour.hpp

@@ -47,6 +47,7 @@ namespace Catch {
 
             Error = BrightRed,
             Success = Green,
+            Skip = LightGrey,
 
             OriginalExpression = Cyan,
             ReconstructedExpression = BrightYellow,

src/catch2/internal/catch_exception_translator_registry.cpp

@@ -44,6 +44,9 @@ namespace Catch {
         catch( TestFailureException& ) {
             std::rethrow_exception(std::current_exception());
         }
+        catch( TestSkipException& ) {
+            std::rethrow_exception(std::current_exception());
+        }
         catch( std::exception const& ex ) {
             return ex.what();
         }

src/catch2/internal/catch_result_type.hpp

@@ -16,6 +16,8 @@ namespace Catch {
         Ok = 0,
         Info = 1,
         Warning = 2,
+        // TODO: Should explicit skip be considered "not OK" (cf. isOk)? I.e., should it have the failure bit?
+        ExplicitSkip = 4,
 
         FailureBit = 0x10,
 

src/catch2/internal/catch_run_context.cpp

@@ -270,6 +270,9 @@ namespace Catch {
         if (result.getResultType() == ResultWas::Ok) {
             m_totals.assertions.passed++;
             m_lastAssertionPassed = true;
+        } else if (result.getResultType() == ResultWas::ExplicitSkip) {
+            m_totals.assertions.skipped++;
+            m_lastAssertionPassed = true;
         } else if (!result.succeeded()) {
             m_lastAssertionPassed = false;
             if (result.isOk()) {
@@ -475,6 +478,8 @@ namespace Catch {
             duration = timer.getElapsedSeconds();
         } CATCH_CATCH_ANON (TestFailureException&) {
             // This just means the test was aborted due to failure
+        } CATCH_CATCH_ANON (TestSkipException&) {
+            // This just means the test was explicitly skipped
         } CATCH_CATCH_ALL {
             // Under CATCH_CONFIG_FAST_COMPILE, unexpected exceptions under REQUIRE assertions
             // are reported without translation at the point of origin.
@@ -571,8 +576,13 @@ namespace Catch {
         data.message = static_cast<std::string>(message);
         AssertionResult assertionResult{ m_lastAssertionInfo, data };
         assertionEnded( assertionResult );
-        if( !assertionResult.isOk() )
+        if ( !assertionResult.isOk() ) {
             populateReaction( reaction );
+        } else if ( resultType == ResultWas::ExplicitSkip ) {
+            // TODO: Need to handle this explicitly, as ExplicitSkip is
+            // considered "OK"
+            reaction.shouldSkip = true;
+        }
     }
     void RunContext::handleUnexpectedExceptionNotThrown(
             AssertionInfo const& info,

src/catch2/internal/catch_test_failure_exception.hpp

@@ -20,6 +20,9 @@ namespace Catch {
      */
     [[noreturn]] void throw_test_failure_exception();
 
+    //! Used to signal that the remainder of a test should be skipped
+    struct TestSkipException{};
+
 } // namespace Catch
 
 #endif // CATCH_TEST_FAILURE_EXCEPTION_HPP_INCLUDED

src/catch2/reporters/catch_reporter_automake.cpp

@@ -17,7 +17,9 @@ namespace Catch {
     void AutomakeReporter::testCaseEnded(TestCaseStats const& _testCaseStats) {
         // Possible values to emit are PASS, XFAIL, SKIP, FAIL, XPASS and ERROR.
         m_stream << ":test-result: ";
-        if (_testCaseStats.totals.assertions.allPassed()) {
+        if ( _testCaseStats.totals.testCases.skipped > 0 ) {
+            m_stream << "SKIP";
+        } else if (_testCaseStats.totals.assertions.allPassed()) {
             m_stream << "PASS";
         } else if (_testCaseStats.totals.assertions.allOk()) {
             m_stream << "XFAIL";

src/catch2/reporters/catch_reporter_compact.cpp

@@ -105,6 +105,11 @@ class AssertionPrinter {
             printIssue("explicitly");
             printRemainingMessages(Colour::None);
             break;
+        case ResultWas::ExplicitSkip:
+            printResultType(Colour::Skip, "skipped"_sr);
+            printMessage();
+            printRemainingMessages();
+            break;
             // These cases are here to prevent compiler warnings
         case ResultWas::Unknown:
         case ResultWas::FailureBit:
@@ -220,7 +225,7 @@ class AssertionPrinter {
 
             // Drop out if result was successful and we're not printing those
             if( !m_config->includeSuccessfulResults() && result.isOk() ) {
-                if( result.getResultType() != ResultWas::Warning )
+                if( result.getResultType() != ResultWas::Warning && result.getResultType() != ResultWas::ExplicitSkip )
                     return;
                 printInfoMessages = false;
             }