Mutations tested unreliably: where to search for the bug?

[kevinmatthes]

I experienced on several different machines with two different code bases that the generated mutations are not always tested reliably. Sometimes, mutations are caught by the same test suite, sometimes not, causing the CI to fail apparently for no reason.

Here is an example:

• here, 9 of 9 mutants are caught; and
• here, 1 of 9 mutants is not caught by the very same test suite.

The above example originates from two subsequent PRs which both do not have changed the test suite. Thus, my question now is: have I called cargo-mutants the wrong way or might this be an edge-case bug?

[kevinmatthes]

Running the failed workflow again seemed to have caught the mutant although I did not have changed the test suite either. To really catch it once and for all, I will need to create another test case.

[sourcefrog]

I don't think you're calling it the wrong way, although you could try adding --no-shuffle to remove one source of variability. (Although arguably it's a feature that shuffling may highlight that there is non-deterministic behavior.)

Also I'd suggest changing your CI to upload mutants.out after the run, like in https://mutants.rs/ci.html, which will help enormously in debugging this kind of thing.

I'd also add -vV to the CI invocation at least temporarily to make it clearer what's happening. (Although this info is also present in the logs.)

What I think is most likely is that one of the tests is not quite deterministic.

If you look in mutants.out/logs at the file for the mutant that is only sometimes caught that should show you which tests are failing sometimes.

[sourcefrog]

OK I could easily reproduce the nondeterminicity here, so it's not something about CI:

Found 9 mutants to test
ok       Unmutated baseline in 2.2s build + 0.1s test
 INFO Auto-set test timeout to 20s
caught   src/tar_archive.rs:58:9: replace TarArchive::create -> Result<()> with Ok(()) in 0.3s build + 0.1s test
caught   src/tar_archive.rs:89:9: replace TarArchive::extract -> Result<()> with Ok(()) in 0.3s build + 0.1s test
caught   src/tar_archive.rs:47:9: replace TarArchive::add_files -> Result<()> with Ok(()) in 0.3s build + 0.1s test
caught   src/tar_archive.rs:77:9: replace TarArchive::exists -> bool with false in 0.3s build + 0.1s test
caught   src/tar_archive.rs:101:9: replace TarArchive::list -> Result<Vec<PathBuf>> with Ok(vec![]) in 0.3s build + 0.1s test
caught   src/tar_archive.rs:125:9: replace TarArchive::remove -> Result<()> with Ok(()) in 0.3s build + 0.1s test
caught   src/tar_archive.rs:132:9: replace TarArchive::update -> Result<()> with Ok(()) in 0.3s build + 0.1s test
caught   src/tar_archive.rs:77:9: replace TarArchive::exists -> bool with true in 0.3s build + 0.1s test
caught   src/tar_archive.rs:101:9: replace TarArchive::list -> Result<Vec<PathBuf>> with Ok(vec![Default::default()]) in 0.3s build + 0.1s test
9 mutants tested in 5s: 9 caught
bobcat ~/src/mutants (main) 07:24
; c r mutants -d ~/src/aeruginous-tbr/ -vV
    Finished dev [unoptimized + debuginfo] target(s) in 0.05s
     Running `target/debug/cargo-mutants mutants -d /home/mbp/src/aeruginous-tbr/ -vV`
Found 9 mutants to test
ok       Unmutated baseline in 2.3s build + 0.1s test
 INFO Auto-set test timeout to 20s
caught   src/tar_archive.rs:77:9: replace TarArchive::exists -> bool with true in 0.3s build + 0.1s test
MISSED   src/tar_archive.rs:125:9: replace TarArchive::remove -> Result<()> with Ok(()) in 0.3s build + 0.1s test
caught   src/tar_archive.rs:89:9: replace TarArchive::extract -> Result<()> with Ok(()) in 0.3s build + 0.1s test
caught   src/tar_archive.rs:101:9: replace TarArchive::list -> Result<Vec<PathBuf>> with Ok(vec![Default::default()]) in 0.3s build + 0.1s test
caught   src/tar_archive.rs:47:9: replace TarArchive::add_files -> Result<()> with Ok(()) in 0.3s build + 0.1s test
caught   src/tar_archive.rs:132:9: replace TarArchive::update -> Result<()> with Ok(()) in 0.3s build + 0.1s test
caught   src/tar_archive.rs:101:9: replace TarArchive::list -> Result<Vec<PathBuf>> with Ok(vec![]) in 0.3s build + 0.1s test
caught   src/tar_archive.rs:58:9: replace TarArchive::create -> Result<()> with Ok(()) in 0.3s build + 0.1s test
caught   src/tar_archive.rs:77:9: replace TarArchive::exists -> bool with false in 0.3s build + 0.1s test
9 mutants tested in 5s: 1 missed, 8 caught

I think I can see in your tests a place where they are not hermetic.

Remember that Rust's test suite (regardless of cargo-mutants) runs multiple tests in parallel. So anything that affects global state of the process or the machine can cause interference between tests.

In this particular case, interestingly, the non-hermicity never seems to cause a problem in normal runs, only under cargo-mutants, and presumably also if you had a naturally-occurring bug.

[kevinmatthes]

and I'm not sure what you mean by the macro-defined tests...

Short answer: https://github.com/kevinmatthes/aeruginous-tbr/blob/734b2dbc8a13237abda6e86c8f557eb5d0586e6c/tests/application.rs#L204-L219.

Long answer: there has been some progress to the repository; there are now 35 tests in total as well as an executable which has four different modes, triggered by various keywords each. To simplify the testing of the keyword mapping to the associated application modes, I wrote a macro (see the link above) which expands to the actual set of tests. These tests do not seem to be executed by the named coverage tools. In another code base, https://github.com/kevinmatthes/aeruginous-rs, I apply even more macros in the tests and there never was any problem -- with none of the coverage tools. Thus, there seems to be something special about that particular repository as multiple tools seem to have difficulties to work with the defined tests.

[kevinmatthes]

The simple fix is to use a temp file or a temp directory, e.g. through the rust tempfile crate.

I will give this a try, thank you for the suggestion!

[kevinmatthes]

Which mutant and which test are you talking about?

Test: https://github.com/kevinmatthes/aeruginous-tbr/blob/734b2dbc8a13237abda6e86c8f557eb5d0586e6c/tests/application.rs#L90-L96

Mutant: src/application.rs:100:9: replace Logic::tar_archive -> Result<()> with Ok(())

The mutant should be caught if there is a test which actually causes the tar_archive method to fail. The test attempts to delete a TAR archive which does not exist, thus the tar_archive method exits with some value other than Ok(()) as the underlying code does so, as well, in case that the archive to be removed does not exist. This mutant is not always caught.

[sourcefrog]

What I would suggest there is:

Make that change by hand in your working tree; see if the tests catch it.

I ran that here from the feature/tar-handling-cli branch and it says all the mutants were missed:

; cargo mutants -f application.rs --regex Logic --in-place
Found 5 mutants to test
ok       Unmutated baseline in 5.2s build + 0.1s test
 INFO Auto-set test timeout to 20s
MISSED   src/application.rs:87:9: replace Logic::resolve_files -> Result<()> with Ok(()) in 0.6s build + 0.1s test
MISSED   src/application.rs:76:31: replace == with != in Logic::main in 0.5s build + 0.1s test
MISSED   src/application.rs:66:9: replace Logic::main -> Result<()> with Ok(()) in 0.6s build + 0.1s test
MISSED   src/application.rs:100:9: replace Logic::tar_archive -> Result<()> with Ok(()) in 0.5s build + 0.1s test

I can also see the tests created a application!application!create!tar_archive.tar file which is an example of the non-hermiticity I talked about.

I ran cargo t --test application and I can see that it runs some tests and then

test application_mode::display::update ... ok
test application_mode::from_str::compress ... ok
test application_mode::from_str::content ... ok
test application_mode::from_str::create ... ok
test application_mode::from_str::decompress ... ok
Interact with Brotli, TAR, TAR.BR, and TBR archives

Usage: tbr [OPTIONS] <MODE> <ARCHIVE> [FILES]...

Arguments:
  <MODE>      The operation to perform on the considered archive
  <ARCHIVE>   The archive to interact with
  [FILES]...  The file(s) to add to the considered archive

Options:
  -d, --destination <DESTINATION>  The directory to unpack the archive's files into; defaulting to the current working directory, if omitted
  -h, --help                       Print help

I think you're calling a Clap function that causes the process to exit(0). If you call this from the tests then, at whatever moment this is hit, it will look like the whole test target was successful.

[kevinmatthes]

Thank you, this solved the problem!