Discussion: speed comparison against Mockito

[stepango]

I'm using Mockito on my current project. Turns out that it's very slow. Would be great to see some benchmarks.

[oleksiyp]

Why are you sure it's mockito is adding slow down?

Anyway, I'll try to benchmark it against mockito. It won't probably be very fast.

The problem here is that the speed is very dependent on use-case.
Do you have your project on GH?
Few things are quite important:

• how much classes do you have?
• how much tests do you have?
• do you have intensive computation involving mock calls?

Class re-transformation definitely adds slow down, but it's cached.
So if you have really big code-base with a lot of classes it can eat some time.
And that is the case for both frameworks

[stepango]

99% sure😁 Bytecode transformations + reflection at runtime is very resource consuming. unfortunately, the project is not public. There are 3k+ tests and most of them are heavily use Mockito. As a result tests execution time ~3min on my machine, which is very long time for unit-tests.

[johnny]

From my (limited) experience, mockk seems to be slow. A single unit tests with one RelaxedMock, one method stub and one verification took more than 1 second to execute with Java 8.

[oleksiyp]

That may be the case. Did you have a chance to compare it to mockito? Im interested to make it faster. Definitely class transformation what makes it slow. If you reuse classes, then it caches it and shouldnt be so slow for subsequent calls. Of course something probably can be improved even now. I understand your case is simple. But anyway if you can share your case as github project I can exactly reproduce it and iterate on performance optimiaztion as far as I can go 29 янв. 2018 г. 2:24 PM пользователь "Jonas von Andrian" < notifications@github.com> написал:

…

From my (limited) experience, mockk seems to be slow. A single unit tests with one RelaxedMock, one method stub and one verification took more than 1 second to execute with Java 8. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#13 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFAbjBl04x7Or67e-IbCQmWVkisYjJtuks5tPcaegaJpZM4RMlcP> .

[gildor]

@johnny I cannot reproduce such slow mocking. RelaxedMock works for me about 2-4ms. Are you sure that you do not measure the first mokk invocation? Because for the first invocation JVM loads classloader, and it's pretty big one in case of mokk (requires 1-1.5 second) but it's the single operation, so +1 second for all your tests instead +1 second on each mock or relaxed mock

[oleksiyp]

No additional information provided, so no possibility to improve. Will get back to it if use case appear

[ratiofu]

Hello @oleksiyp! I pulled a test for a component I recently wrote into a public repo: https://github.com/ratiofu/Utf8StreamReader - compare the startup and run time of Utf8StreamReaderTest (using MockK) to a similar test I just wrote for comparison, NonMockedUtf8StreamReaderTest. I am not so much concerned with a comparison to Mockito, but more with the duration of unit tests per se. The mocked test has between 7 to 8 seconds of startup time before it actually executes–independent of whether I run it with Gradle or from within IntelliJ. The project from which I pulled this has a lot of dependencies, but the test seems to have about the same startup time whether I run it in its original project or in the small project I shared above.

[oleksiyp]

@ratiofu interesting observation. it is happening not always I believe, something probably is special about this test. Can you send me log output on TRACE level for this test? To have logs you need to add SLF4J, logback and provide logback-test.xml in resources. Example here and here

[ratiofu]

@oleksiyp, attached please find the trace (the Github project I shared above is updated accordingly). Longest delay is here:

615 [main] TRACE io.mockk.impl.JvmMockKGateway - Starting Java MockK implementation. Java version = 1.8.0_152.
5980 [main] TRACE io.mockk.proxy.MockKInstrumentation - Byte buddy agent installed
(logger is configured to show time since start)

mockk-trace.log

[oleksiyp]

Yep looks like ByteBuddy installation takes a lot of time. You can try running agent in command line if you would like to compare. Anyway worth to check ByteBuddy issues if there is some bug that cause delay under some conditions. I have no access to PC during day, so will get back to the issue in the evening/weekends. Thanks 9 марта 2018 г. 8:49 AM пользователь "Thomas Jung" <notifications@github.com> написал:

…

@oleksiyp <https://github.com/oleksiyp>, attached please find the trace (the Github project I shared above is updated accordingly). Longest delay is here: 615 [main] TRACE io.mockk.impl.JvmMockKGateway - Starting Java MockK implementation. Java version = 1.8.0_152. 5980 [main] TRACE io.mockk.proxy.MockKInstrumentation - Byte buddy agent installed (logger is configured to show time since start) mockk-trace.log <https://github.com/oleksiyp/mockk/files/1795985/mockk-trace.log> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#13 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFAbjBXXal-5lEtZ7oa1YvICyqo0iuJTks5tcjQFgaJpZM4RMlcP> .

[oleksiyp]

@raphw can you sugest why ByteBuddy can be slow at startup?

[raphw]

The installation is a very expensive operation in itself, unfortunately. It often requires a stop the world with full garbage collection. It is especially expensive since Java 9 where Byte Buddy needs to do some extra tricks to get around the restriction. I'd recommend to profile a run and see what takes the most time, it is very machine dependant but I can have a look if something is pointed out.

The installation relies a bit on being aone time operation, it does therefore work best for running big suits rather then single tests. It is also more costly on Windows machines, by my experience.

To avoid the expensive start up, you can add byte-buddy-agent as a JVM javaagent pameter. Byte Buddy will detect thatt and not run the expensive routine.

[raphw]

Just some general feedback: advice instances are immutable and very expensive. Dont create them for each transformation. Your synchronized maps will also slow things down significantly.

[oleksiyp]

Thanks! 9 марта 2018 г. 2:31 PM пользователь "Rafael Winterhalter" < notifications@github.com> написал:

…

Just some general feedback: advice instances are immutable and very expensive. Dont create them for each transformation. Your synchronized maps will also slow things down significantly. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#13 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFAbjHx_kLfx3AqEyNKAczovqj7iUl7Tks5tcoQ5gaJpZM4RMlcP> .

[angryziber]

Unfortunately, mockk is significantly slower than mockito when creating mocks:

for (i in 1..100000) mock<Date>() - 1082 ms (default mockito)
for (i in 1..100000) mock<Date>() - 2029 ms (with mock-maker-inline)
for (i in 1..100000) mockk<Date>() - 3523 ms

Why mockk is so much slower?

[angryziber]

More measurements for hot-cases only (first mock creation is not measured):
default mockito - 356 ms
inline mockito - 876 ms
mockk - 1657 ms

[oleksiyp]

@angryziber what is JDK used for measurements?

[oleksiyp]

I think there is a lot of things to consider regarding how to do benchmark itself. Can you share some code snippet to make my life easier?

Mockk creates subclasses together with inlining to make abstract methods mockable. I wonder if that can be the reason of slow down. Another trick is that for instantiation second subclass is created(subclass itself is cached, but one-time subclass creation is performed). Do not remember all the details on why it is needed. I think this can have more fine-grained control when it is applied.

[oleksiyp]

Following code:

fun main(args: Array<String>) {
    var t = System.currentTimeMillis()
    ByteBuddyAgent.install()
    ByteBuddy().subclass(Date::class.java).make().load(Thread.currentThread().contextClassLoader)
    println(System.currentTimeMillis() - t)
    t = System.currentTimeMillis()
    for (i in 1..100000) mockk<Date>()
    println(System.currentTimeMillis() - t)
    t = System.currentTimeMillis()
    for (i in 1..100000) mockk<Date>()
    println(System.currentTimeMillis() - t)
    t = System.currentTimeMillis()
    for (i in 1..100000) mockk<Date>()
    println(System.currentTimeMillis() - t)
}

Gives me following answer:

550
1825
532
378

[raphw]

If you subclass and retransform, you can pretty much sum up up the numbers for both on the Mockito side and that seems right, too. Retransformation is expensive as it discards a lot of optimizer work, there is not much to improve here.

[oleksiyp]

@raphw my only point is that subclassing not always needed and can be skipped depending if a class has abstract methods or not.

[lwasyl]

I'll share my measurements as well, although unfortunately they're not very structured. I've converted a small project, roughly 100 tests across 6 modules, from Mockito to Mockk. I then ran Gradle Profiler with default options and test tasks set to never be up to date, to ensure that it's almost exclusively tests run that's measured. Here are the results for Mockito, with mock-maker-inline applied where required (durations are in milliseconds):

And here are for Mockk:

In the first Mockito version I have mock-maker-inline applied to 4/6 modules, but if I apply it to all of them (including a module with 25% of all tests), the results are a bit closer:

Unfortunately I'm not able to investigate more and give more specific profiling results, but it does seem like general byte-buddy performance issue. Thing is, with Mockito I can at least disable mock-maker-inline in modules where I don't need it, and overall I'd say not needing it is (or should be) the norm, not the exception. I'd love to have an option to opt-in to more expensive mocking on a per-case basis, when it happens that I'm mocking an external class.

Another thing people sometimes do is they ditch mock-maker-inline in favor of all-open compiler plugin. For some this might be preferable, since it's a small one-time cost, although afaik works only for own classes.

[oleksiyp]

Thanks for comparation. Mockk will lost most of features without inlining

[raphw]

Instrumentation is a very expensive feature. Byte Buddy needs to jump through some hoops to get it done. You can try to add byte-buddy-agent as a Java agent on the command line if you want to avoid the majority of costs. The good news anyways is thst the costs are one-time. In big test suits, the costs amortize.

[lwasyl]

The good news anyways is thst the costs are one-time. In big test suits, the costs amortize.

I'm more concerned about running individual tests or test classes from IDE during normal development. Adding javaagent:byte-buddy-agent.jar does help, but still, statistics for a small test class with 12 tests, 2 mocks (just one of final class):

time [s]	library	byte-buddy-agent applied
3.5 - 4.0	Mockk	no
3.0 - 3.2	Mockk	yes
1.5 - 1.6	Mockito	no
1.2 - 1.3	Mockito	yes

Mockk is still two times slower. And in this case the costs unfortunately don't amortize, because such test classes are run in isolation many times.

I'm not trying to argue which library is better or whether the speed is acceptable, just sharing my stats and thoughts. In any case some automated benchmark and comparison between Mockito, Mockito with mock-maker-inline and Mockk would help track performance and make informed decision about mocking library. I personally love the API, but to me the speed is a dealbreaker.

[raphw]

It would be interesting to attach a profiler to both Mockk and the Mockito runs to see what the time is spent on.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. If you are sure that this issue is important and should not be marked as stale just ask to put an important label.

[angryziber]

What about putting an important label as not suggests?

…

On Wed, Aug 28, 2019, 21:23 stale[bot] ***@***.***> wrote: This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. If you are sure that this issue is important and should not be marked as stale just ask to put an important label. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#13?email_source=notifications&email_token=AAGXKYGSCQG6QAEPM4VCH6TQG27CDA5CNFSM4EJSK4H2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5MBALQ#issuecomment-525865006>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGXKYEZMFOFD27F3YF4QGDQG27CDANCNFSM4EJSK4HQ> .

[oleksiyp]

@raphw btw this initialization time depends on some conditions, right? Alike is it JVM, JDK. Which VM version is used. Which OS is used. Wont it be nice to have benchmark and suggest some particular version in docs.

[raphw]

Of course, newer Java versions tend to be faster as they do more initialization lazier. Generally, absolute benchmark numbers are seldom meaningful, just comparison numbers of the same run; therefore it is often good to also add a no-op benchmark as a baseline.

[kaushalyap]

Any work going on to improve performance?

[tmszdmsk]

I spent some time preparing the JMH testing rig for mockk (#763). I did profile a simple case of creating a mock and stubbing single method and here you can find Java Flight Recorder profiling results: 92c448c#diff-6cd22ac6a6f83bf3ea8124c66fd49373dcd1f28bad5e30354d5fc13c880fdf36.

Two observations (all % below are relative to "total time" (it's sampling profiler so it's not really time)):

1. setting up mock

every { mockedClass.mockedFun() } returns "Hello, mockk!"

seems to take up more time (58%) than actually creating a mock(41%) with

val mockedClass: MockedClass = mockk()

2. use of ConcurrentHashMap (via WeakMockHandlersMap) seems to be the major contributor to mock creation (34%). It is non-insignificant part of mock setup as well.

Could we drop using ConcurentHashMap in favour of non-thread-safe implementation? At least in my case mockk is not used that often in a multi-threaded scenarios. Maybe separate mode could be implemented?

[oleksiyp]

Hi, @tmszdmsk thanks for the work done on benchmarking MockK.

In my opinion, there are few things:

• it is not possible to remove the multithreading part from MockK as hundreds or thousands of projects already rely on these possibilities
• tbh I doubt there is any problem with ConcurrentHashMap usage

Few questions:

• have you taken into account that on the first call to mockk() ByteBuddy is initializing for 1-2seconds? Can you remove this initialization from your tests if it is present(by calling mockk() before tests)?
• ConcurrentHashMap is I believe also create only once, so can this also be taken into account?
• what are the absolute duration numbers of mock creation and setting up via every

Again thank you for your contribution, just want us to get to all roots regarding these questions, so happy to drill down and assist.

[raphw]

You can reduce the initialization time by a lot by:

• allowing for self attachment for agents
• adding the byte buddy agent jar as a javaagent to your VM
  Otherwise, BB has to work around the attachment restrictions, unfortunately, what costs this time.

[tmszdmsk]

Hi @oleksiyp, @raphw, thanks for your replies.

it is not possible to remove the multithreading part from MockK as hundreds or thousands of projects already rely on these possibilities

Yes, I can imagine. I was thinking about additional mode when creating mocks (like threadSafe = false) where it would switch the implementation used.

tbh I doubt there is any problem with ConcurrentHashMap usage

that's possible, but this is based on the data I collected, you can easily downlaod the JFR(Java Flight Recorder) file and review it yourself, IntelliJ has native support for .jfr files.

have you taken into account that on the first call to mockk() ByteBuddy is initializing for 1-2seconds? Can you remove this initialization from your tests if it is present(by calling mockk() before tests)?

I don't think this skews the results that much. I think JMH takes care of those one-off events by running "warmup" iterations before the actual benchmark where it warms the JVM.

ConcurrentHashMap is I believe also create only once, so can this also be taken into account?

It's not ConcurrentHashMap creation that shows up in the samples, it's put that is the most "popular".
Below is the mock creation part of the whole flamegraph (the wider it is the more times it appeared in collected samples ~ the most time it takes)

what are the absolute duration numbers of mock creation and setting up via every

JFR doesn't provide such numbers as it is sampling profiler (checking the status of the JVM every X ms and producing the results based on that). Hard to tell.

From my experience with property-based tests (which cause many iterations of the same test to be executed) it's every & verify blocks that have the biggest impact on the test time execution. I think it matches the observation from JFR where single every block takes "more time" than mock creation (image below)

@raphw, it is possible to attach the agent in the benchmark execution (as it is simple jar file), but I didn't have time to do it. My hypothesis based on above is that it won't make much (if any) difference.

I think good next step would be to actually quickly "hack" using different implementation of the map just to run those tests again and compare the results. I may find some time to do this sometime this week, but maybe someone could look into it earlier (and has more experience with mockk codebase). I am happy to help if anyone needs it.

[sharmando]

@raphw

You can reduce the initialization time by a lot by:

allowing for self attachment for agents
adding the byte buddy agent jar as a javaagent to your VM
Otherwise, BB has to work around the attachment restrictions, unfortunately, what costs this time.

Do you have any resources that would help me implement such workarounds?

[raphw]

Here is a reference: https://stackoverflow.com/questions/50498102/how-to-set-jdk-attach-allowattachself-true-globally

You can also find byte-buddy-agent on Maven Central and add that jar via -javaagent: to your test VM.

[aSemy]

There's a couple of small updates on improving MockK performance

Benchmarking

There are now basic benchmark tests in the project #904 (credit to @tmszdmsk for providing them). They're a good baseline. They show that it still the case that creating a stub is expensive , even if the stub is not used.

Here's a graph comparing no stubbing or mocking (which is obviously very fast), only mocking (which is of course slower), and mocking and creating a stub (which, as noted previously, is significantly slower)

.

It would be a huge benefit to improve the benchmarks. Here's what I'd like to see:

1. wider test coverage (for example of spy{} and answers{})
2. Automatic testing - add a new GitHub action for running the benchmark tests.
3. Tracking performance over time. I don't know what the best way to achieve this is. Is there a website, like codecov?

Help would be really appreciated!

Dropping ConcurentHashMap

Could we drop using ConcurentHashMap in favour of non-thread-safe implementation? At least in my case mockk is not used that often in a multi-threaded scenarios. Maybe separate mode could be implemented?

I experimented with replacing the ConcurrentHashMap, but I didn't notice any significant improvements :(

isValue

The new isValue check seems to be very expensive (I had a look at a flamegraph but forgot to take a screenshot)

mockk/modules/mockk/src/jvmMain/kotlin/io/mockk/impl/recording/JvmSignatureValueGenerator.kt

Line 19 in 7d0ae19

if (cls.isValue) {

MockK + KSP

I've been thinking about updating MockK to take advantage of the new Kotlin kapt replacement: KSP

Other Kotlin mocking frameworks (https://github.com/bitPogo/kmock, https://github.com/mockative/mockative) use KSP to replace reflection, and they can do this in multiplatform projects. The drawback is that both libraries only mock non-sealed interfaces - they don't create 'random' instances of objects (presumably for multiplatform compatibility reasons).

I wonder if MockK can use a hybrid approach, and use KSP to introspect the source code during compilation, save the results to disk, and then during runtime instead of using reflection (which is expensive), determine which 'random' object to create using the saved data.

I think this 'compile time reflection' approach is similar to what https://github.com/JetBrains-Research/reflekt aims to do, but progress on that project seems slow, and has restrictions on Kotlin versions.

[Raibaz]

I just did a quick google search, how about [this])(https://github.com/boswelja/kotlinx-benchmark-table-action) to integrate kotlinx-benchmark in a GHA?

Also, KSP looks really interesting, and I agree we could take a hybrid approach by doing symbol processing at compile time and then using the result only if needed.

It would basically be moving some complexity (and time) from runtime to compile time, but I think it's worth looking into.