Global fixtures

[keirlawson]

I have a requirement to run some tests in parallel (as they are slow end-to-end tests) however I'd like them to share resources. So far as I can see this is not presently possible with munit as there is no mechanism for fixtures shared by test suites.

[olafurpg]

Thank you for reporting! It would be fantastic if we could add support for global fixtures.

As discussed on Discord, one way to accomplish this is to extend munit.Framework and update testFrameworks := to use your custom framework implementation. However, I am concerned this will not work when running MUnit tests via IntelliJ. Another risk with that approach is that we initialize expensive global fixtures even when they're not needed (for example, because you're running testOnly MySuite that doesn't use that fixture).

Another way to accomplish this would be to register global fixtures by overriding a method name override def munitGlobalFixtures = List(...), similar to how munitFixtures works. We probably want to make global fixtures async by default (see #418). This might be cleaner approach because

• it's consistent with local fixtures
• it avoids initializing expensive global fixtures when they're not needed
• probably works out of the box in Scala.js and Native

[olafurpg]

After a closer look I am concerned this is more tricky than I originally expected. It's easy to acquire global resources but it's tricky to know when to close them.

One workaround you can try today is to put the global resources on a Scala object and register a JVM shutdown hook. Make sure to enable the sbt setting fork := true so that the shutdown hook gets executed. One downside with this approach is the tests will succeed even if an exception gets thrown during shutdown. Here's a complete example

object MyGlobalFixture {
  private var i = 0
  def message(): String = {
    i += 1
    s"message${i}"
  }
  Runtime.getRuntime
    .addShutdownHook(new Thread {
      override def run(): Unit = {
        println(s"SHUTTING DOWN ${i}")
        throw new RuntimeException("BOOM") // gets ignored
      }
    })
}

class GlobalFixtureSuite1 extends munit.FunSuite {
  test("fixture1") {
    println(MyGlobalFixture.message())
  }
}

class GlobalFixtureSuite2 extends munit.FunSuite {
  test("fixture2") {
    println(MyGlobalFixture.message())
  }
}

The output from running both tests

sbt:munit> testsJVM/Test/testOnly munit.GlobalFixtureSuite1 munit.GlobalFixtureSuite2
message1
munit.GlobalFixtureSuite1:
  + fixture1 0.063s
message2
munit.GlobalFixtureSuite2:
  + fixture2 0.0s
SHUTTING DOWN 2
Exception in thread "Thread-1" java.lang.RuntimeException: BOOM
  | => tat munit.MyGlobalFixture$$anon$1.run(GlobalFixtureSuite.scala:15)
[info] Passed: Total 2, Failed 0, Errors 0, Passed 2

If we want to avoid JVM shutdown hooks then it gets trickier because MUnit only has visibility into the execution of a single test suite. We can easily fix this problem for sbt since we control the test framework implementation, but I'm struggling to find a solution that would also work in IntelliJ, Gradle, and Maven.

One workaround within the JUnit framework is to write a test suite in Java that aggregates several MUnit suites like this

// AllSuites.java (I wasn't able to get this working with Scala)
@RunWith(org.junit.runners.Suite.class)
@org.junit.runners.Suite.SuiteClasses({
  MUnitSuite1.class,
  MUnitSuite2.class
})
public class AllSuites {
  @ClassRule public static StringResource myString = new StringResource();
}

The field AllSuites.myString will then get correctly closed after all of the listed suites finish execution. However, this means you must always run all of those tests together, I don't think you can easily run an individual test case.

[olafurpg]

@keirlawson would you be able to try out the workaround above and report if it works for your codebase?

[olafurpg]

JUnit 5 has APIs that might make it possible to implement global fixtures, see ExtensionContext.Store.CloseableResource (https://junit.org/junit5/docs/5.3.0/api/org/junit/jupiter/api/extension/ExtensionContext.Store.CloseableResource.html). If the JVM shutdown hook works then we could design an MUnit API today that

• does the correct thing for sbt (since we control the testing framework) and works for JVM, Scala.js and Scala Native
• uses JVM shutdown hooks in IntelliJ/Gradle/Maven
• can migrate to JUnit 5 CloseableResource in the future to avoid JVM shutdown hooks in non-sbt clients

[olafurpg]

Here's a draft of a global fixture API that we could expose in MUnit

trait GlobalFixture[A] {
  def apply(): A
  def beforeAllTestSuites(): Future[Unit]
  def afterAllTestSuites(): Future[Unit]
}

You would then register global fixtures just like regular fixtures

  abstract class FunSuite {
    def munitFixtures: List[Fixture[_]]
+   def munitGlobalFixtures: List[GlobalFixture[_]]
  }

  object db extends GlobalFixture[DatabaseConnection] { ...  }
  class MySuite extends FunSuite {
+    override val munitGlobalFixtures = List(db)
    test("connect-db") {
+     db().query("...")
    }
  }

[keirlawson]

@keirlawson would you be able to try out the workaround above and report if it works for your codebase?

I wasn't able to get this working for my particular code, however I suspect that was more to do with cats-effect Runtime weirdness than the approach not in principal being viable

[olafurpg]

@keirlawson did you enable fork := true?

[keirlawson]

Yep

[olafurpg]

@keirlawson Do you have a minimized example with cats-effect that I could try to reproduce?

[Daenyth]

Not sure how helpful it is, but weaver-test implements something like this. Their architecture is very different though so I'm not sure if the design translates

[olafurpg]

@Daenyth the biggest challenge is coming up with a solution that works in all clients (IntelliJ, Gradle, Maven, sbt). This requirement means we need to somehow work within the JUnit framework. There’s nothing in JUnit 4 that allows this from what I can tell but it’s easy to implement with JUnit 5 “TestFramework”.