Exception thrown by forked test framework swallowed

[retronym]

steps

Clone https://github.com/retronym/sbt-test-fork-swallows-error and follow instructions in the README.

sbt> test
[error] Uncaught exception when running tests: java.lang.Throwable
[trace] Stack trace suppressed: run last test:test for the full output.
Exception in thread "Thread-4" java.io.EOFException
    at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2626)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1321)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
    at sbt.React.react(ForkTests.scala:114)
    at sbt.ForkTests$$anonfun$mainTestTask$1$Acceptor$2$.run(ForkTests.scala:74)
    at java.lang.Thread.run(Thread.java:745)
[success] Total time: 0 s, completed 31/08/2016 2:36:09 PM

problem

The exception thrown by the tasks method of the Runner is logged to console, shipped over the wire to the host SBT process, logged again, but the task result is still success.

expectation

The test task should fail.

notes

Discovered in the scala/scala SBT build which uses the partest implementation of the SBT test interface.

[eed3si9n]

Thanks for the repro project. Probably a duplicate of #2442

[retronym]

I've been unable to come up with a workaround. How could I assert that >0 tests were executed?

[eed3si9n]

According to #2442 (comment) it's a bug in sbt.
There's an if expression that checks >0 tests are about to get executed here -

sbt/main/actions/src/main/scala/sbt/ForkTests.scala

Lines 23 to 26 in 16a6906

	if (opts.tests.isEmpty)
	constant(TestOutput(TestResult.Passed, Map.empty[String, SuiteResult], Iterable.empty))
	else
	mainTestTask(runners, opts, classpath, fork, log, config.parallel).tagw(config.tags: _*)

[retronym]

This can be used to fail the test execution if there is no events. I don't know if this test will give false positives, though.

executeTests in Test := {
  val result = (executeTests in Test).value
  if (result.events.isEmpty) {
    (streams.value.log.error("No test events found"))
    result.copy(overall = TestResult.Error)
  }
  else result
}

[eed3si9n]

@retronym Thanks for the hint. Here's my fix - #2723

sbt-test-fork-swallows-error> test
[error] Uncaught exception when running tests: java.lang.Throwable
[trace] Stack trace suppressed: run last test:test for the full output.
[error] Error: Total 0, Failed 0, Errors 0, Passed 0
[error] Error during tests:
[error]         Forked test harness failed: null
[error] (test:test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 1 s, completed Aug 31, 2016 4:18:57 AM

[dwijnand]

Fixed in #2730 by @eed3si9n