Data sources for tests (C++)

[jl-wynen]

It would be great if we had a general, customisable way of generating inputs for tests.

At the moment, most tests only use a single or a few handcrafted Variables and DataArrays. Those are good for probing special cases. But we should increase test coverage by adding ways to generate more data (semi-) automatically.

One can use explicit loops in the test cases to iterate over a large space of inputs.

• + Customisable.
• + Can cover whole product space of parameters.
• - Completely manual (could be slightly improved by having data generators outside of tests).
• - Doesn't produce structured output.
• - Bloats the test code.

Use a general fixture for related tests and instantiate it on multiple different sets of parameters

(see #1488)

• + Set up once, use many times.
• + Integrated into gtest -> produces useful progress / failure reports.
• + Has 'parameter generators' that can create ranges, Cartesian products, and more.
• - No integration of random number generators. Could seed one globally and use to instantiate fixtures but there would be no good way of recording its state to inspect and replay failed tests.
• - Blanket set of parameters for all test cases, cannot exclude parameters that don't work in a particular test.
• - Not composable (I think), e.g. how can we generate datasets from variables?
• - Cannot combine type- and value-parameterised tests (I think).
• - Generators are still configured manually and ignorant of code to test, cannot adapt.

Test frameworks that can generate input data intelligently to probe the code for weaknesses

• Not sure if anything exists that can integrate with gtest. Fuzzers are probably overkill and would want to operate on their own executable.

The gtest fixture option looks best to me right now. But we need to think of ways of building a good corpus of test data. We also should see if there is a way to skip unusable combinations of parameters in a test that is not too verbose.

[SimonHeybrock]

See also #513 for original discussion.

[SimonHeybrock]

I think I mostly agree with your analysis. More context to also keep in mind:

1. We have and are moving to generic code to reduce specific testing efforts.
   • transform-based ops mainly get their kernel tested.
   • By using what has been implemented in Configure files for scipp functions using cmake #1443 in more cases we can completely eliminate code in variable/ and dataset/ ofr certain functions, further reducing testing need.
2. Data sources as discussed above cannot and will not fully replace test-by-test custom data (with the exception of certain tests). I see it more as something additional, as a tool to ensure that we do not forget "regions of the parameter space" when implementing new functionality.