~30s pause when running tests

[charleskorn]

I'm not sure if this is something in my setup or an issue in Spek, but when I run the tests in https://github.com/charleskorn/batect with ./gradlew app:test, Gradle seems to pause for nearly 30 seconds at :app:test > 0 tests completed before then starting to run my tests.

Do you have any ideas on what the cause of this might be?

Running the tests with a profiler suggests that most of this time is spent calling resolveSpec for each test class.

[raniejade]

That usually means you have some long running logic during the discovery phase. i.e

describe("something") {
  runLongRunningThing()

  it("some test") {
    ...
  }
}

[raniejade]

Probably this is the culprit? https://github.com/charleskorn/batect/blob/master/app/src/journeyTest/kotlin/batect/journeytests/VersionInfoJourneyTest.kt#L32

[charleskorn]

Hey @raniejade, thanks for the quick response! ./gradlew :app:test only runs the tests in https://github.com/charleskorn/batect/tree/master/app/src/unitTest (the journey test you linked to there runs quite quickly).

[charleskorn]

Looking at this more closely, it looks like the time is spent loading classes and waiting for Mockito to generate mocks. Would you be interested in a PR that runs the spec resolution in parallel? (I'm imagining something like replacing the map call here with a call to the pmap described at https://jivimberg.io/blog/2018/05/04/parallel-map-in-kotlin/)

[raniejade]

Is mockito thread-safe though?

[charleskorn]

Yes: https://github.com/mockito/mockito/wiki/FAQ#is-mockito-thread-safe

[raniejade]

Nice, I'm currently working on integrating coroutines internally - let's hold this for now until I'm done with it.

[charleskorn]

Hey @raniejade, are you still working on integrating coroutines? I'm still keen to have a crack at this

[raniejade]

Hey @charleskorn! Yes, you find it at #835. Looks like everything is working, just need to do some testing since it is quite a big change.

[charleskorn]

OK great, I'll keep an eye on that PR then.

[raniejade]

@charleskorn fyi, #835 is now merged!

[arturbosch]

Hello @raniejade ,

we recently splitted some of our code to new modules to reduce incremental build and test time.
However we noticed some increase in test discovery since then.
It gets even worse when using --parallel. Modules with ~10 tests take 20 seconds to be discovered (detekt/detekt#2902).
In some modules (rules) we have some heavy setup code but in others the tests are plain simple and still hang for many seconds before starting the work.

I've migrated two simple modules to junit jupiter and the test discovery improvement is huge - detekt/detekt#2902 (comment)!

We rely heavily on Spek with more than 1700 tests. Do you have any idea how we can improve the startup time? Thanks!

[raniejade]

@arturbosch Thanks for the report! I suspect this line

spek/spek-runtime/src/jvmMain/kotlin/org/spekframework/spek2/runtime/JvmDiscoveryContextFactory.kt

Line 48 in 60c14fe

if (testDirs.isNotEmpty()) {

is causing the whole classpath to be scanned. I'll try making the scan asynchronous

spek/spek-runtime/src/jvmMain/kotlin/org/spekframework/spek2/runtime/JvmDiscoveryContextFactory.kt

Line 52 in 60c14fe

return cg.scan().use {

- hopefully it will help. I will add this in my todo list this weekend.

[arturbosch]

Thanks for the quick reply and the ongoing effort! It will help a lot.
Would it be possible to restrict the scanning to some user defined packages?
This could potentially also improve the performance.

[raniejade]

Would it be possible to restrict the scanning to some user defined packages?

I can look at that, but I don't think Gradle's implementation of JUnit 5 support is able to pass that info: https://github.com/gradle/gradle/blob/a9f9b343a3ebfa34534d8245908a1f5702271ac9/subprojects/testing-junit-platform/src/main/java/org/gradle/api/internal/tasks/testing/junitplatform/JUnitPlatformTestClassProcessor.java#L117

From what I can see, if you set a test filter in gradle - gradle itself will do the filtering and passed in classes (I'm not sure of the selection criteria here) to JUnit Platform launcher.