Skip to content

Commit

Permalink
Only reseed the internal RNG when a test is first entered
Browse files Browse the repository at this point in the history
This fixes multiple issues with random generators, with the most
important one being that multiple nested generators could return
values from the same sequence, due to internal implementation
details of `GENERATE`, and how they interact with test case
paths.

The cost of doing this is that given this simple `TEST_CASE`,
```cpp
TEST_CASE("foo") {
    auto i = GENERATE(take(10, random(0, 100));
    SECTION("A") {
        auto j = GENERATE(take(10, random(0, 100));
    }
    SECTION("B") {
        auto k = GENERATE(take(10, random(0, 100));
    }
}
```

`k` will have different values between running the test as
a whole, e.g. with `./tests "foo"`, and running only the "B"
section with `./tests "foo" -c "B"`.

I consider this an acceptable cost, because the only alternative
would be very messy to implement, and add a lot of brittle and
complex code for relatively little benefit.

If this calculation changes, we will need to instead walk
the current tracker tree whenever a random generator is being
constructed, check for random generators on the path to root,
and take a seed from them.
  • Loading branch information
horenmar committed May 17, 2022
1 parent 7a2a6c6 commit dcafc60
Showing 1 changed file with 33 additions and 2 deletions.
35 changes: 33 additions & 2 deletions src/catch2/internal/catch_run_context.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -193,6 +193,39 @@ namespace Catch {
assert(rootTracker.isSectionTracker());
static_cast<SectionTracker&>(rootTracker).addInitialFilters(m_config->getSectionsToRun());

// We intentionally only seed the internal RNG once per test case,
// before it is first invoked. The reason for that is a complex
// interplay of generator/section implementation details and the
// Random*Generator types.
//
// The issue boils down to us needing to seed the Random*Generators
// with different seed each, so that they return different sequences
// of random numbers. We do this by giving them a number from the
// shared RNG instance as their seed.
//
// However, this runs into an issue if the reseeding happens each
// time the test case is entered (as opposed to first time only),
// because multiple generators could get the same seed, e.g. in
// ```cpp
// TEST_CASE() {
// auto i = GENERATE(take(10, random(0, 100));
// SECTION("A") {
// auto j = GENERATE(take(10, random(0, 100));
// }
// SECTION("B") {
// auto k = GENERATE(take(10, random(0, 100));
// }
// }
// ```
// `i` and `j` would properly return values from different sequences,
// but `i` and `k` would return the same sequence, because their seed
// would be the same.
// (The reason their seeds would be the same is that the generator
// for k would be initialized when the test case is entered the second
// time, after the shared RNG instance was reset to the same value
// it had when the generator for i was initialized.)
seedRng( *m_config );

uint64_t testRuns = 0;
do {
m_trackerContext.startCycle();
Expand Down Expand Up @@ -422,8 +455,6 @@ namespace Catch {
m_shouldReportUnexpected = true;
m_lastAssertionInfo = { "TEST_CASE"_sr, testCaseInfo.lineInfo, StringRef(), ResultDisposition::Normal };

seedRng(*m_config);

Timer timer;
CATCH_TRY {
if (m_reporter->getPreferences().shouldRedirectStdOut) {
Expand Down

0 comments on commit dcafc60

Please sign in to comment.