Consider ability to fail a test if it completes with a non-unit value

[mpilquist]

The design of munit makes integration with effect libraries very simple. Test bodies have type => Any and handlers are for various types are registered via ValueTransforms and TestTransforms. The munit-cats-effect and scalacheck-effect libraries each take advantage of this capability. The former registers value transforms for IO and SyncIO values and the latter registers a value transform for PropF.

One downside to this approach is that failing to register the appropriate transform (e.g., by forgetting to mixin CatsEffectSuite), results in tests that pass without fully evaluating the body. For example, the following test always passes because there's no transform telling munit how to evaluate IO values:

class MySuite extends FunSuite {
  test("this should fail") {
    IO(fail("oh no"))
  }
}

One way to fix this in user land is by writing a test transform that requires every test to return a Unit value:

package munit

import scala.concurrent.Future

trait StrictSuite extends FunSuite {
  override def munitTestTransforms: List[TestTransform] =
    super.munitTestTransforms ++ List(munitValidateOnlyUnits)

  private val munitValidateOnlyUnits: TestTransform =
    new TestTransform(
      "Validate only units",
      t =>
        t.withBodyMap[Future[Any]] {
          _.map {
            case ()    => ()
            case other => fail(s"Test resulted in a non-unit value: $other")
          }(munitExecutionContext)
        }
    )
}

Of course, this must be mixed in to be effective so perhaps it's not worth it. This also could be annoying with assertions that return a value -- e.g., intercept

[olafurpg]

Thank you for reporting! Can you elaborate on the expected behavior of such a feature? Would it be enabled through a test option? I'm not opposed to adding such an option, although it's unlikely we could enable this by default since it would cause many valid tests to fail.

One possible approach would be to expose hooks for subclasses of munit.Framework 🤔 This would allow you to programmatically define how to handle test values and you can enable it via sbt with testFrameworks := Seq(new TestFramework("my.custom.StrictMUnitFramework")).

This also could be annoying with assertions that return a value -- e.g., intercept

We could probably special-case Throwable alongside Unit.