automatic regression (threshold computation and comparison) for any metrics

[matthieu-nesme]

Feature Request

context

CDash offers a the test timing feature where it computes a threshold based on stats from previous execution timings and compare the current execution timing to check if it is in an acceptable range.
It's a cool feature to detect wall-clock timing regressions.

request

It would be even cooler to be able to detect such regressions for any metrics.

For instance, I would activate it for CPU-time and RAM & VRAM consumption.

suggestions

Indeed, in CTest can report extra test measurements to CDash.
For each measurement defined in the CDash project, let's propose if an automatic threshold computation and comparison should be performed.

• Only for numeric/ measurements.
• The measurement columns in the viewTest.php page could give a status.
• The graph associated to a measurement could draw the threshold line.
• what about the main page of a project index.php? Maybe an extra column per tested measurement? Similar to the "Time" column.

---

How can we make CDash better?

That is great to see lots of improvments these last few months, thanks!

[williamjallen]

Great suggestion! There are a few things coming down the pipeline such as the new GraphQL API which should make this easier to implement. I think it would be a great addition, but we'd have to find a funding source.