Allow sharded execution for bazel test sharding support

[ChewyGumball]

Description

This PR adds support for splitting test execution across multiple invocations of the test binary (otherwise known as test sharding). The tests to run are split evenly into the number of groups given by the new Config::shardCount(), and the group at the index given by the new Config::shardIndex() is executed. The intended use is that multiple processes will be run, each with the same shard count, but different shard indices. Sharding works for executing as well as listing tests, and is applied after the list of tests to execute has been created (after filters).

The main motivation is to allow Bazel test sharding to work:

#define CATCH_CONFIG_RUNNER
#include "catch.hpp"

#include <cstdlib>
#include <filesystem>

int main( int argc, char* argv[] )
{
  Catch::Session session;

  int returnCode = session.applyCommandLine( argc, argv );
  if( returnCode != 0 )
        return returnCode;
 
  // writing to session.configData() or session.Config() here 
  // overrides command line args
  // only do this if you know you need to

  char* envShardCount = std::getenv("TEST_TOTAL_SHARDS");
  if (envShardCount != nullptr && session.configData().shardCount != 1) {
        session.configData().shardCount = std::stoul(envShardCount);
  }

  char* envShardIndex = std::getenv("TEST_SHARD_INDEX");
  if (envShardIndex != nullptr && session.configData().shardIndex != 0) {
        session.configData().shardIndex = std::stoul(envShardIndex);
  }

  char* envShardStatusFile = std::getenv("TEST_SHARD_STATUS_FILE");
  if (envShardStatusFile != nullptr) {
      std::filesystem::last_write_time(envShardStatusFile, std::chrono::file_clock::now());
  }

  int numFailed = session.run();
  
  // numFailed is clamped to 255 as some unices only use the lower 8 bits.
  // This clamping has already been applied, so just return it here
  // You can also do any post run clean-up here
  return numFailed;
}

[codecov]

Codecov Report

Merging #2257 (6ac20a0) into devel (905bf43) will decrease coverage by 0.08%.
The diff coverage is 79.17%.

@@            Coverage Diff             @@
##            devel    #2257      +/-   ##
==========================================
- Coverage   91.06%   90.98%   -0.08%     
==========================================
  Files         150      151       +1     
  Lines        7139     7187      +48     
==========================================
+ Hits         6501     6539      +38     
- Misses        638      648      +10     

[horenmar]

Hi, didn't have time to go through this sooner.

I like the general idea, but there is a bunch of changes to be done before merging.

[ChewyGumball]

All great comments. I've tried my best to address them in the latest commit. Would you mind taking another look?

[horenmar]

I probably won't get to the tests till tomorrow.

[horenmar]

Thanks, it looks pretty good now, but I think you went overboard on reducing the test script.

What I would want to see is:

• Pick a positive tag (generators is fine)
• List all tests for that tag without sharding
• Shard the tests into say, 5 shards, list each shard individually, check that they add up into the whole list, and without overlap.
• Shard the tests into the same numbers of shards, check that executing each shard executes the tests listed in previous step.

Since the expensive part is calling into the binary, rather than doing extra processing of the data on the Python side, it is possible to also add extra checks along the way, e.g. checking that the binary obeys --order selection when sharding and so on, because that only adds some extra processing in Python.