Allow specifying alerts on metrics different than CPU

[pmalek]

Currently, when specifying the alert threshold e.g. alert-threshold: '200%' we cannot specify the metric which we want to alert on.

For instance Go toolchain reports the following metrics:

• ns per benchmark
• Bytes allocated per benchmark
• allocations per benchmark

This action compares the first metric - ns per benchmark - which might be flaky e.g. on some CI systems.

This issue is a feature request to allow alerts on different metrics.

[miparnisari]

This action compares the first metric - ns per benchmark - which might be flaky e.g. on some CI systems.

hey @pmalek, i'm not an owner of this repo but I have a Go project and AFAIK this is not true. This action compares all 3 metrics that you mentioned. Example: https://github.com/openfga/openfga/actions/runs/7302898086

Also, why would ns/op be flaky? You need to make sure that in your setup you compare benchmarks across same CPUs.

[pmalek]

Also, why would ns/op be flaky? You need to make sure that in your setup you compare benchmarks across same CPUs.

I'm running the benchmarks on Github provided runners and those seem to differ across runs

[miparnisari]

I'm running the benchmarks on Github provided runners and those seem to differ across runs

Ah yes, that is true. Since you are publishing raw results maybe you need to publish them per CPU 🤔 🤷‍♀️ I don't know