Slow on large codebase

[sourcefrog]

[joshparallel]

This seems like a great approach to limit the combinatoric explosion of possible mutation tests in large codebases. 👍 Is it straightforward to implement? E.g. I've been running cargo mutants on my company's codebase and I'm currently seeing

about 525 min remaining

😅

[sourcefrog]

Hi Josh,

Actually, I think the specific thing of testing only each mutant's own package was already fixed since 1.0.0.

Maybe something is going wrong with this. Could you have a look at some of the files in mutants.out/log and see if it seems to be running only the relevant tests for each mutant, or, otherwise doing anything really strange?

The other package related thing to fix is to allow generating mutants for only one package at a time, although this can be approximated today by using --file patterns.

I'd like to understand a bit more about the tree where you're seeing that long ETA. If you run cargo mutants --list does it seem to be generating an excessive number of mutants for each function, maybe due to complex return types? How long would the tests on this tree and the packages within it normally take?

If you have a machine with a good amount of RAM compared to how much the tests use and a lot of cores you could try -j which might reduce the overall time.

[joshparallel]

@sourcefrog Ah I see, thanks for clearing up my confusion. I see 7099 mutants having been generated, and running all our Rust tests takes ~400 s:

Found 7099 mutants to test
Unmutated baseline ... ok in 185.3s build + 401.1s test

I'm currently attempting to run with -j 8 and it's projecting to take quite a while!

72/7099 mutants tested, 1% done, 26 missed, 26 caught, 20 unviable, 22:26 elapsed, about 1096 min remaining

But maybe that's a pretty reasonable number given the size of the codebase and the intrinsic complexity of needing to run each test within a package once for every mutation within that package. Am I understanding correctly?

[ASuciuX]

Hi @joshparallel ! I've experienced similar challenges with large packages leading to prolonged build times. In my case, mutants take between 100 to 600 seconds to build, and tests require around 1300 seconds. Additionally, I had to configure test-thread=1 to ensure the tests run correctly in my tests so that increased the time also. With around 12,000 mutants to handle, this seems to be the expected timeframe for such large packages.

[sourcefrog]

@joshparallel Yes, for each generated mutant it needs to do an incremental build and then (if that succeeds) run cargo test until it finds a failure. So in the worst case, with 7099 mutants and a 400s test suite that would be 32 days, quite a lot! In practice some of the mutants will be unviable (won't build), and some tests will fail faster, so not quite that much but still significant.

My later experiments seem to show that setting -j too high can cause thrashing, so about -j2 or -j3 seems to be the sweet spot, though it will depend on your tree and hardware.

Anyhow, I recently added a --shard option that lets you split work across multiple VMs, and this can probably help a lot because it's very parallelizable. For example if you launched 50 VMs you should be able to cut the total time by nearly 50.

There is also a new feature to test only the code changed in a PR, which does not completely remove the need to test all the code but should make it more reasonable to run from CI on each PR.

@ASuciuX

Additionally, I had to configure test-thread=1 to ensure the tests run correctly in my tests so that increased the time also.

I'm pretty sure this is due to your crate having non-hermetic tests or dependencies. Most people won't need this.