Fail on no tests found?

[rlbisbe]

In specnaz the following code results in "no test found" and a sucesful build.

import org.specnaz.junit.SpecnazJUnit;

public class TestSpec extends SpecnazJUnit {
    {
        describes("Test spec", it -> {
        });
    }
}

This is the IntelliJ output:

Process finished with exit code 0
Empty test suite.

If we run it as part of a bigger test set, the result is similar.

Process finished with exit code 0

Should we make the test fail if the discovery phase results in zero tests?

For reference, if you add a test in JUnit and you mark it as ignored, it's effectively the same.

public class TestTest {

    @Test
    @Ignore
    public void doTest(){

    }
}

The output for this method is the following.

Test ignored.
Process finished with exit code 0

[ivoanjo]

👍 I also run into this issue -- I copy pasted my test code in the describes while experimenting with the library, but then was seeing that no tests run on the output, which was a bit confusing.

[skinny85]

I've done some research on this topic, and what I found is that the behavior is different between JUnit and TestNG.

The following code:

import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;

@RunWith(JUnit4.class)
public class ExampleTest {
}

fails a build with a "no runnable methods" error - the BlockJUnit4ClassRunner validates whether a class has any test methods (see code here).

However, this code:

import org.testng.annotations.Test;

@Test
public class ExampleTest {
}

works completely fine in TestNG, and does not cause the build to fail.

I think the Specnaz integrations should behave similarly to the framework their integrating with - so, my instinct is to fail the build in the JUnit Runner if no tests were found, but ignore it in the TestNG Factory.

Thoughts on this approach?