-
Notifications
You must be signed in to change notification settings - Fork 240
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support incremental analysis #322
Comments
With Karma using Chrome, I am seeing scan times for ~3,500 mutants take ~5 hours. Using PhantomJS decreases scan times by ~50%. Incremental analysis while not perfect would go a long way in improving performance while still killings a majority of the generated mutants. |
Yeah, this would be pretty crucial. Stryker works well in my very small apps. My main prod app has ~1k unit tests. I had to reduce runners to 6 to allow it to run, and think after a few hours it was less than 50% done. You could probably go two ways with this one- just running on changes from that commit, or using a history file like pitest for Java. |
Wow this is cool. Are you using coverage analysis for the running of the tests? If so, is it OK to share some statistics? How much faster is it with/without it. Also #366 could also make a big difference for these kind of projects.
What exactly do you mean with "scan times". Would that be the initial test run by any chance? Another way of solving this is with sharding. So you could create a build per file to be mutated. You could share the same stryker.conf.js file, but choose to mutate different files using an env variable and parameterised build. We could force stryker to merge reports together in a big report. It might also be a good idea to split your project into different modules all together, but that's not up to me ;)
@jmpresso How does it work exactly? Does every developer need to build the history themselves using a full run once? Or is there a centralized store somewhere? |
Not sure exactly what you mean there. It was taking so long to run I gave up on letting it finished. So currently I only have the istanbul type line coverage. I think karma runs in <2 minutes by itself, the code coverage is pretty fast too. I do agree #366 would help as well. The temporary folder was also very large during my run.
I'm not totally sure how much that would help. Are you suggesting sharding it and running the shards in parallel? It might help, but might bog down the build machine. Perhaps we could make it work with SauceLabs. We'd also have to figure out different ports for all the different Karma servers, so it might get pretty messy. I agree... modularizing absolutely helps, it's just not always possible.
Only the first time using the plugin. From there, it records the results in the history file, which could then gets versioned along with the code. I'm honestly not sure if/how it would handle merges and what not though. FWIW, it mutates in memory also. |
Wouw, 2 mins. Not sure there is something we can do in that case. Are there slow tests in the mix? Or just in general a large application footprint? If you have slow tests, you could try to move them to a separate test suite (for example: "integration tests"). If you have a big application foodprint, it could help to bundle files together. For example: 1 bundle for all tests and one bundle for your production code. It might be that the majority of the time has to do with loading files in the browser. HTTP 1.1 is terribly bad optimized for a big number of files.
No, running the shards one by one, or on multiple build servers.
@jmpresso Just to be sure: you wouldn't check in the history file (or files) into source control? |
@nicojs Sorry for the confusion, by "scan times", I meant "execution time". Similar to @jmpresso the project I was referring to executes 5,000+ tests in 4 minutes using just karma. However, when executing Stryker with multiple runners, it takes 4+ hours. I think #366 will help too. However, the biggest benefit seems like it would come from a history file feature like PiTest uses. This way previously generated mutants' results are used for files not modified since the last build. This history file IS committed to source control and referenced in the maven config settings in the pom.xml. This way the CI system can see it during builds and mutants generated in unmodified code are not rechecked again. Developers can keep this file up to date by running PiTest locally, generating a new pitestHistory.txt and committing it to source control. In the example below if FooClass is not updated on a subsequent commit, PiTest will use its previous results from the pitestHistory.txt file instead of regenerating its mutants and running tests against them. Example class... package path.to.src.class;
import path.to.src.class.BarClass;
import path.to.src.class.BazClass;
public class FooClass {
public BazClass methodFromClass(BarClass barClass) {
return Object.doStuff(BarClass)
}
} Resulting entry in pitestHistory.txt after running PiTest... 1
<classHistory><id hierarchicalHash="f9bf32"><classId hash="16367410"><name><name>path/to/src/class/FooClass</name></name></classId></id><coverageId>0</coverageId></classHistory>
<result><id><location><clazz><name>path/to/src/class/FooClass</name></clazz><method><name>methodFromClass</name></method><methodDesc>(Lpath/to/src/class/BarClass;)Lpath/to/src/class/BazClass;</methodDesc></location><indexes><int>7</int></indexes><mutator>org.pitest.mutationtest.engine.gregor.mutators.ReturnValsMutator</mutator></id><status numberOfTestsRun="0" status="NO_COVERAGE"><killingTest class="org.pitest.functional.Option$None"/></status></result> |
It is also described http://pitest.org/quickstart/incremental_analysis/ |
I created a fork of Stryker in order to begin working on this feature. After cloning the Stryker project and running When i run |
@CarlosRosario Thanks for wanting to take a shot at this! Could you try running |
@simondel that is what @CarlosRosario did ( As far as I can tell it should work. We run this command on linux and windows machines to develop and build stryker. Maybe something was not installed correctly? Can you try clearing your node_modules and try again? After running If you use vscode, you can open a child package in vscode and use Please contact us on gitter if you have any problems or questions. |
My bad! I do notice that your fork could be using typescript 2.8, which we don't support yet. Although the error is different then the one we got with typescript 2.8. You could try to clone the core repo instead of your fork to see if you can build/test that. |
Hey, any news/plans about this? |
I don't think Carlos is working on it any longer. I know one of us made a fork to work on it, and made some progress to at least creating the history file, but then it got put on hold. I think there also is still a lot of work needed to migrate to the new version of Stryker, and finish the implementation. It's possible it will get picked back up at some point, but not sure. I know we have also been pushing to start using faster test runners like Jest over Karma, to try and reduce the need. TL/DR, unless the Stryker team is working on it, probably not, as I don't think some of us here that have been discussing it have made significant progress on it. |
We're not actively working in it. We've also decided to start implementing #1514 since that will give us a lossless performance boost. This might be something we would consider doing after that. |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
Dear bot, don't close this, please |
I'm closing this issue in favor of #2753 |
Remove `Mutant.range` from the plugin API. Plugin creators should rely on `location` instead (which has `start` and `end` positions). See changes in the TypeScript checker for an example. The `range` property was a way to identify changed code. It was a tuple defined as `[start: number, end: number]`. As plugin creator, you had the opportunity to use this instead of `location`. We've chosen to remove it because having 2 ways to identify location is redundant and this will allow us to implement #322 in the future. BREAKING CHANGE: The `range` property is no longer present on a `mutant`. Note, this is a breaking change for plugin creators only. Co-authored-by: Simon de Lang <simondelang@gmail.com>
The mutation testing process could be sped up by only testing changed mutants.
We could argue that a mutant should not have to be tested if:
I would not advise to run incremental analysis during your build because it will probably not be flawless. One of the flaws of my proposed setup is that the test(s) that killed the mutant in the previous run could have changed. Right now we have no good way of knowing they were changed.
The text was updated successfully, but these errors were encountered: