False positive result on Sat4j 2018/10/25 experiment

[assadOW2]

Characteristics

• Issue Type: [test report]
• Reproducibility: [always]
• Severity: [feature]
• Execution Environment: [ubuntu 16.04]
• Reporter: [Assad Montasser, am@ow2.org]

Description

This refers to this output: https://github.com/STAMP-project/descartes-usecases-output/tree/master/OW2/Sat4j/201810251534/issues/org.sat4j.tools.RunnableSolver

When manually commenting all the run() method, original test suite detects a time out, resulting in a failure. This is due to the fact that without onFinishWithAnswer() running, there is an infinite loop.

The method should not appear in Descartes output since this is not a real survivor.

Steps to reproduce

Other files and URLs

Relationships

Help on issue template

Preview to follow the link or open file .github/ISSUE_DOC.md

[assadOW2]

I ran Descartes after the last commit on Sat4j: https://gitlab.ow2.org/sat4j/sat4j/commit/309b935d53606fcba2f4e009da82590c993a857d

This commit was aiming to cover the createClause() method in https://github.com/STAMP-project/descartes-usecases-output/tree/master/OW2/Sat4j/201810251534/issues/org.sat4j.minisat.constraints.MixedDataStructureDanielWLConciseBinary

The new results are : https://github.com/STAMP-project/descartes-usecases-output/tree/master/OW2/Sat4j/201811081117

Results show that not only the targeted method is resolved, but also this one ( run() ) is no more in the report. But now, we have the onFinishWithAnswer() in the output (which was not detected previously)

https://github.com/STAMP-project/descartes-usecases-output/tree/master/OW2/Sat4j/201811081117/issues/org.sat4j.tools.ManyCore

[danielleberre]

The infinite loop also occurs when removing the body of onFinishWithAnswer(). It is also a false positive.

[oscarlvp]

Could you check that the test cases that detect the methods in the original test suite are also executed when running Descartes?
If not all the test cases were executed there might be a chance that the test case that detects the method was left out.
The list of test cases executed is at the bottom of the report for every method.
In the cases of run and onFinishWithAnswer they both seem to be covered only by org.sat4j.BugSAT34.testUnitClause.

[oscarlvp]

@assadOW2 any news on this? Did you try the previous solution?

[assadOW2]

Hi @oscarlvp !
I checked the two last outputs, https://github.com/STAMP-project/descartes-usecases-output/tree/master/OW2/Sat4j/201811120812 and https://github.com/STAMP-project/descartes-usecases-output/tree/master/OW2/Sat4j/201811151111, Descartes reports show that both run and onFinishWithAnswer original tests were executed.