xwiki-platform-store-filesystem-oldcore is flickering #127
Comments
|
PIT's log on that module for failing build https://ci.xwiki.org/job/xwiki-platform_pitest/626/consoleFull |
|
PIT's log for the previous successful build on the same module. https://ci.xwiki.org/job/xwiki-platform_pitest/625/consoleFull |
|
@oscarlvp cool, is that what you meant by the pitest logs? I thought you were talking about files in the target/ directory. If the console logs are enough I can make sure to include them in the issue reports. Thx |
|
@vmassol No, no, I meant the files in the target directory, but I'm using these logs as an approximation, I'm comparing them, just now I was writing the next comment. |
Yeah it's better than nothing... |
|
The difference between both builds lay in the For the successful build: Notice how there are 15 So it seems that this is another case where the difference is due to |
|
After executing the analysis twice the following mutants
This produces a variation in the mutation score. The only test executing these mutants is which uses mocks. We have seen flickering in other modules with similar tests using mocks and covering code that manipulates files. There were 35 As an example, the following mutants were
But as they are considered as detected in both executions, then they do not affect the score. |
|
@vmassol Did you decide what to do with this issue based on the previous information? |
|
Hi @oscarlvp. Thanks for pinging me. Actually I looked again at your comments yesterday. From what I see the next step is to be able to find the new test source code generated by a mutation and run it manually to find out why it takes long to execute, right? |
|
@vmassol remember that the mutation will only change the code of the method not the code of the test. In this case, reproducing the mutation is simple. In the first table from the previous comment you can see the methods and the mutations. |
|
@oscarlvp Sorry my bad, I wrote it too fast without thinking, yes I meant the mutation of the sources (not the test). Ok I need to be able to reproduce locally first and then find where it's stored in the target dir but that should be easy. My question was: is that what's needed now to progress or did you have something else in mind? Thx! |
|
@vmassol Yes, I think this should reveal the actual effect of the mutation and it requires a good understanding of the code base. |
|
@oscarlvp so I'm not exactly sure what I should do. Here's what I did:
In both cases the tests succeeded and they executed in less than 2 seconds. What am I missing? Thanks |
|
@vmassol the issue may not happen all the time, as there is some flickering. Also you have to compare the time of the mutant with respect to the time of the original code. PIT sets the timeout of the mutant using the time of the original code as reference. |
I really don't understand how there can be some flickering here TBH. So I've modified the method to always return false and executed
I don't understand this part. The time of the original code is when that code was modified so it could have been like 10 years before. Also what is the default timeout? From what I see it's 4s right? At https://pitest.org/quickstart/maven/ it says: We don't override timeoutConstant and we use 1.25 for timeoutFactor so I guess our timeout is 1.25 * 4000 = 5s ?
Sure, I agree, but that's not the point here. The poins is to have pitest/descartes be stable and don't fail the build for wrong reasons. What can I try more? Thanks! |
I meant execution time for the original code without mutation and the mutated code when the tests are executed. You can check the actual implementation here If you say that the mutated code does not delay the test execution that much, then my guess (a wild guess) is that something in the code instrumented by PIT is causing the issue. Maybe it has to be with the mocks since these test do use them and similar tests have shown the same timeout instability in the past. PIT does a couple of weird things to support mocks, but I don't know for sure if that is the real issue. |
See #62 for the context.
(happened at https://ci.xwiki.org/job/xwiki-platform_pitest/626/console)
@oscarlvp you asked for the pitest report but I don't have access to it when a pitest execution flickers since it happes on the CI and usually I can't reproduce locally. Also since we run job in transient docker containers, I don't have the target directory output, it's discarded. To get it, we would need to modify our generic pipeline code to save it somehow.
Tried to reproduce locally and couldn't with
mvn clean install -Pquality -Dxwiki.pitest.skip=false.The text was updated successfully, but these errors were encountered: