feat: functionality to bound the execution

[yxliang01]

Current design of Vertigo will attempt to generate all possible mutants in the defined mutation space, only then report will be generated. However, it is likely that there is no enough budget to perform the evaluation of all possible mutants. For example, it can take ~20 mins to run the test suite of OpenZeppelin. So, it's impractical to wait for the testing task to complete for the report. As a result, a functionality to limit the execution boundary might be a good idea, and maybe also make Vertigo generate report at exit (potentially unexpected) as long as one mutant was generated and evaluated so that external tool like timeout can be used to bound the execution timeout.

Thanks

[JoranHonig]

You can use --sample-ratio to use the mutant sampling technique. In which case you take a random subset of the mutations to run analysis on. This improves runtime, and decreases the precision of the reported mutation score.

I also thought I merged a feature that allows you to use a non-random partition of the mutations.
This helps with determinism, or when you want to do the analysis in separate runs.
I will try to find this code and merge it.

and maybe also make Vertigo generate report at exit (potentially unexpected) as long as one mutant was generated and evaluated so that external tool like timeout can be used to bound the execution timeout.

I like this idea 👍

[yxliang01]

@JoranHonig Indeed, I overlooked this option. However, it's not intuitive to set a wanted value though I think. Since the number of mutants Vertigo can generate various from one combination of versions of project under test and Vertigo and Vertigo doesn't support output the number of mutants possible currently (then, terminate) if I'm not mistaken. I feel normal user would prefer setting the number of mutants to be generated instead, since then the running time should then be predictable.

[RickGriff]

It would be great if you could select 1) the contracts to mutate and 2) the test file(s) to run.

For systems that are fairly modular and loosely coupled, this could save a lot of time/resources.

[JoranHonig]

@RickGriff you can filter directories from execution . Example: --exclude contracts/mock

Let me think about better ways to do this. Technically incremental mutation testing could help a lot already.