Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cross-references and a single spy cause memory leak #1533

Closed
arturdryomov opened this issue Nov 15, 2018 · 6 comments
Closed

Cross-references and a single spy cause memory leak #1533

arturdryomov opened this issue Nov 15, 2018 · 6 comments

Comments

@arturdryomov
Copy link

Not entirely sure, but I think that Mockito does not handle cross-references well. I have a semi-complex sample that proves this, but at the same time I’m not sure it is a Mockito failure and especially a fixable one. Suggestions how to avoid this behavior in general will be very helpful!

BTW I can provide a .hprof file if you are interested.

Versions

org.mockito:mockito-core:2.22.0
org.mockito:mockito-inline:2.22.0
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)

Gradle

Heap is set to 64 MB.

tasks.withType<Test> {
    maxHeapSize = "64m"
    jvmArgs("-XX:+HeapDumpOnOutOfMemoryError")

    failFast = true
}
$ ./gradlew :module:cleanTestDebugUnitTest :module:testDebugUnitTest --tests "com.github.sample.SubscriptionMemoryLeakSpec"

Code

package com.github.sample

import com.jakewharton.rxrelay2.BehaviorRelay
import com.jakewharton.rxrelay2.PublishRelay
import io.reactivex.disposables.CompositeDisposable
import org.jetbrains.spek.api.Spek
import org.jetbrains.spek.api.dsl.it
import org.junit.platform.runner.JUnitPlatform
import org.junit.runner.RunWith
import org.mockito.Mockito

@RunWith(JUnitPlatform::class)
class SubscriptionMemoryLeakSpec : Spek({

    repeat(1_000) { iteration ->

        it("iteration $iteration") {
            // Remove Mockito.spy and OOM disappears (even without component.unbind).
            val service = Mockito.spy(Service())
            val memoryConsumingService = MemoryConsumingService()

            val component = Component(service, memoryConsumingService)

            component.bind()

            // Uncomment the following line and OOM disappears.
            // component.unbind()
        }

    }

}) {

    class Service {
        val stream = PublishRelay.create<Unit>().toSerialized()
    }

    class MemoryConsumingService {
        // See at as a mass to fill the RAM.
        val streams = (0..1_000).map { BehaviorRelay.create<Int>() }
    }

    class Component(
            private val service: Service,
            private val memoryConsumingService: MemoryConsumingService
    ) {

        private val disposable = CompositeDisposable()

        fun bind() {
            disposable += service.stream.subscribe {
                // This closure keeps a reference to Component.
                memoryConsumingService.streams.size
            }
        }

        fun unbind() = disposable.clear()
    }

}
> Task :module:testDebugUnitTest
java.lang.OutOfMemoryError: GC overhead limit exceeded
Dumping heap to java_pid31753.hprof ...
Heap dump file created [96660586 bytes in 0.332 secs]

com.github.sample.SubscriptionMemoryLeakSpec > it iteration 260 STANDARD_ERROR
    java.lang.OutOfMemoryError: GC overhead limit exceeded
    	at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.<init>(ReentrantReadWriteLock.java:338)
    	at java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync.<init>(ReentrantReadWriteLock.java:669)
    	at java.util.concurrent.locks.ReentrantReadWriteLock.<init>(ReentrantReadWriteLock.java:240)
    	at java.util.concurrent.locks.ReentrantReadWriteLock.<init>(ReentrantReadWriteLock.java:230)
    	at com.jakewharton.rxrelay2.BehaviorRelay.<init>(BehaviorRelay.java:99)
    	at com.jakewharton.rxrelay2.BehaviorRelay.create(BehaviorRelay.java:77)
    	at com.github.sample.SubscriptionMemoryLeakSpec$MemoryConsumingService.<init>(SubscriptionMemoryLeakSpec.kt:41)
*** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message can't create byte arrau at JPLISAgent.c line: 813
    	at com.github.sample.SubscriptionMemoryLeakSpec$1$1$1.invoke(SubscriptionMemoryLeakSpec.kt:21)
    	at com.github.sample.SubscriptionMemoryLeakSpec$1$1$1.invoke(SubscriptionMemoryLeakSpec.kt:14)
    	at org.jetbrains.spek.engine.Scope$Test.execute(Scope.kt:102)
    	at org.jetbrains.spek.engine.Scope$Test.execute(Scope.kt:80)
    	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:105)
    	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$82/1204954813.execute(Unknown Source)
    	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:72)
    	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:95)
    	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:71)
    	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService$$Lambda$89/1263089654.accept(Unknown Source)
    	at java.util.ArrayList.forEach(ArrayList.java:1257)
    	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
    	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:110)
    	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$82/1204954813.execute(Unknown Source)
    	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:72)
    	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:95)
    	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:71)
    	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService$$Lambda$89/1263089654.accept(Unknown Source)
    	at java.util.ArrayList.forEach(ArrayList.java:1257)
    	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
    	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:110)
    	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$82/1204954813.execute(Unknown Source)
    	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:72)
    	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:95)
    	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:71)

com.github.sample.SubscriptionMemoryLeakSpec > it iteration 260 FAILED
    java.lang.OutOfMemoryError


261 tests completed, 1 failed
> Task :module:testDebugUnitTest FAILED
82 actionable tasks: 5 executed, 77 up-to-date

Eclipse Memory Analyzer

screen shot 2018-11-15 at 11 57 52

screen shot 2018-11-15 at 11 58 42

screen shot 2018-11-15 at 11 59 17

screen shot 2018-11-15 at 11 59 58

screen shot 2018-11-15 at 12 01 25


Seems like this happens:

  • Service is a spy and is being passed to Component along with MemoryConsumingService.
  • Since Service is a spy it is being held by Mockito.
  • Component subscribes to Service.stream. The subscribe closure captures Component, Service and MemoryConsumingService. Due to RxJava specifics Service.stream holds all of these thanks to the closure.
  • Since Service is being held by Mockito (since it is a spy) and by itself (due to the subscribe closure) Mockito does not release it.

However:

  • Releasing the subscription via component.unbind() removes a reference, so Mockito releases it and there is no OOM.
  • Avoiding making Service a spy eliminates OOM as well, i. e. cross-references are not an issue for the JVM itself.
@sergejsha
Copy link

The issue is most likely caused by 2 factors:

  1. CreationSettings.spiedInstance holds a hard reference to spied instance.

  2. InlineByteBuddyMockMaker puts instance of the settings into a hard-referenced value of mocks field.

A solution could be to make CreationSettings.spiedInstance a weak reference field.

@arturdryomov
Copy link
Author

@TimvdLippe, @bric3, 🏓

@mockitoguy
Copy link
Member

@ming13, @beworker, does anyone of you wants to take a stab at it? Can you write up a short design note how you would like to solve it and pros/cons of adding another weak reference field? Thanks so much!

@arturdryomov
Copy link
Author

@mockitoguy, hey, thanks for the reply! I think I don’t have enough expertise to do this without breaking something along the way. But just to confirm — is the issue really with Mockito or it is just a bad practices leading to this behavior?

ttanxu added a commit to ttanxu/mockito that referenced this issue Feb 12, 2019
Due to the introduction of map from weak reference from mock instance to
its invocation handler, Mockito became vunerable to memory leaks as
there are multiple situations where Mockito could unintentionally hold
strong references to mock instances in the map record. The strong
references could be through spiedInstance for spies, and arguments used
to faciliate method stubbing.

Mockito could never know if the arguments passed in for method stubbing
are also strongly referenced somewhere else or not, so Mockito needs to
save a strong reference to these arguments to avoid premature GC.
Therefore to solve cyclic strong references through arguments Mockito
needs to explicitly know when mocks are out of their lifespan, and clean
up all internal strong references associated with them.

This commit also fixes mockito#1532 and fixes mockito#1533.
ttanxu added a commit to ttanxu/mockito that referenced this issue Feb 12, 2019
Due to the introduction of map from weak reference from mock instance to
its invocation handler, Mockito became vunerable to memory leaks as
there are multiple situations where Mockito could unintentionally hold
strong references to mock instances in the map record. The strong
references could be through spiedInstance for spies, and arguments used
to faciliate method stubbing.

Mockito could never know if the arguments passed in for method stubbing
are also strongly referenced somewhere else or not, so Mockito needs to
save a strong reference to these arguments to avoid premature GC.
Therefore to solve cyclic strong references through arguments Mockito
needs to explicitly know when mocks are out of their lifespan, and clean
up all internal strong references associated with them.

This commit also fixes mockito#1532 and fixes mockito#1533.
ttanxu added a commit to ttanxu/mockito that referenced this issue Feb 12, 2019
Due to the introduction of map from weak reference from mock instance to
its invocation handler, Mockito became vunerable to memory leaks as
there are multiple situations where Mockito could unintentionally hold
strong references to mock instances in the map record. The strong
references could be through spiedInstance for spies, and arguments used
to faciliate method stubbing.

Mockito could never know if the arguments passed in for method stubbing
are also strongly referenced somewhere else or not, so Mockito needs to
save a strong reference to these arguments to avoid premature GC.
Therefore to solve cyclic strong references through arguments Mockito
needs to explicitly know when mocks are out of their life, and clean
up all internal strong references associated with them.

This commit also fixes mockito#1532 and fixes mockito#1533.
ttanxu added a commit to ttanxu/mockito that referenced this issue Feb 12, 2019
Due to the introduction of map from weak reference from mock instance to
its invocation handler, Mockito became vunerable to memory leaks as
there are multiple situations where Mockito could unintentionally hold
strong references to mock instances in the map record. The strong
references could be through spiedInstance for spies, and arguments used
to faciliate method stubbing.

Mockito could never know if the arguments passed in for method stubbing
are also strongly referenced somewhere else or not, so Mockito needs to
save a strong reference to these arguments to avoid premature GC.
Therefore to solve cyclic strong references through arguments Mockito
needs to explicitly know when mocks are out of their life, and clean
up all internal strong references associated with them.

User can call MockitoSessionBuilder#trackAndCleanUpMocks() to let the
session track and clean up mocks. It also relies on implementation of
mock makers to cleanUpMock(Object). SubclassByteBuddyMockMaker is
a intentional no-op to avoid unnecessary behavior changes in stable
features.

This commit also fixes mockito#1532 and fixes mockito#1533.
ttanxu added a commit to ttanxu/mockito that referenced this issue Feb 12, 2019
Due to the introduction of map from weak reference from mock instance to
its invocation handler, Mockito became vunerable to memory leaks as
there are multiple situations where Mockito could unintentionally hold
strong references to mock instances in the map record. The strong
references could be through spiedInstance for spies, and arguments used
to faciliate method stubbing.

Mockito could never know if the arguments passed in for method stubbing
are also strongly referenced somewhere else or not, so Mockito needs to
save a strong reference to these arguments to avoid premature GC.
Therefore to solve cyclic strong references through arguments Mockito
needs to explicitly know when mocks are out of their life, and clean
up all internal strong references associated with them.

User can call MockitoSessionBuilder#trackAndCleanUpMocks() to let the
session track and clean up mocks. It also relies on implementation of
mock makers to cleanUpMock(Object). SubclassByteBuddyMockMaker is
a intentional no-op to avoid unnecessary behavior changes in stable
features.

This commit also fixes mockito#1532 and fixes mockito#1533.
ttanxu added a commit to ttanxu/mockito that referenced this issue Feb 12, 2019
Due to the introduction of map from weak reference from mock instance to
its invocation handler, Mockito became vunerable to memory leaks as
there are multiple situations where Mockito could unintentionally hold
strong references to mock instances in the map record. The strong
references could be through spiedInstance for spies, and arguments used
to faciliate method stubbing.

Mockito could never know if the arguments passed in for method stubbing
are also strongly referenced somewhere else or not, so Mockito needs to
save a strong reference to these arguments to avoid premature GC.
Therefore to solve cyclic strong references through arguments Mockito
needs to explicitly know when mocks are out of their life, and clean
up all internal strong references associated with them.

User can call MockitoSessionBuilder#trackAndCleanUpMocks() to let the
session track and clean up mocks. It also relies on implementation of
mock makers to cleanUpMock(Object). SubclassByteBuddyMockMaker is
a intentional no-op to avoid unnecessary behavior changes in stable
features.

This commit also fixes mockito#1532 and fixes mockito#1533.
ttanxu added a commit to ttanxu/mockito that referenced this issue Feb 13, 2019
Due to the introduction of map from weak reference from mock instance to
its invocation handler, Mockito became vunerable to memory leaks as
there are multiple situations where Mockito could unintentionally hold
strong references to mock instances in the map record. The strong
references could be through spiedInstance for spies, and arguments used
to faciliate method stubbing.

Mockito could never know if the arguments passed in for method stubbing
are also strongly referenced somewhere else or not, so Mockito needs to
save a strong reference to these arguments to avoid premature GC.
Therefore to solve cyclic strong references through arguments Mockito
needs to explicitly know when mocks are out of their life, and clean
up all internal strong references associated with them.

User can call MockitoSessionBuilder#trackAndCleanUpMocks() to let the
session track and clean up mocks.

This commit also fixes mockito#1532 and fixes mockito#1533.
ttanxu added a commit to ttanxu/mockito that referenced this issue Feb 28, 2019
Due to the introduction of map from weak reference from mock instance to
its invocation handler, Mockito became vunerable to memory leaks as
there are multiple situations where Mockito could unintentionally hold
strong references to mock instances in the map record. The strong
references could be through spiedInstance for spies, and arguments used
to faciliate method stubbing.

Mockito could never know if the arguments passed in for method stubbing
are also strongly referenced somewhere else or not, so Mockito needs to
save a strong reference to these arguments to avoid premature GC.
Therefore to solve cyclic strong references through arguments Mockito
needs to explicitly know when mocks are out of their life, and clean
up all internal strong references associated with them.

This commit also fixes mockito#1532 and fixes mockito#1533.
ttanxu added a commit to ttanxu/mockito that referenced this issue Feb 28, 2019
Due to the introduction of map from weak reference from mock instance to
its invocation handler, Mockito became vunerable to memory leaks as
there are multiple situations where Mockito could unintentionally hold
strong references to mock instances in the map record. The strong
references could be through spiedInstance for spies, and arguments used
to faciliate method stubbing.

Mockito could never know if the arguments passed in for method stubbing
are also strongly referenced somewhere else or not, so Mockito needs to
save a strong reference to these arguments to avoid premature GC.
Therefore to solve cyclic strong references through arguments Mockito
needs to explicitly know when mocks are out of their life, and clean
up all internal strong references associated with them.

This commit also fixes mockito#1532 and fixes mockito#1533.
ttanxu added a commit to ttanxu/mockito that referenced this issue Feb 28, 2019
Due to the introduction of map from weak reference from mock instance to
its invocation handler, Mockito became vunerable to memory leaks as
there are multiple situations where Mockito could unintentionally hold
strong references to mock instances in the map record. The strong
references could be through spiedInstance for spies, and arguments used
to faciliate method stubbing.

Mockito could never know if the arguments passed in for method stubbing
are also strongly referenced somewhere else or not, so Mockito needs to
save a strong reference to these arguments to avoid premature GC.
Therefore to solve cyclic strong references through arguments Mockito
needs to explicitly know when mocks are out of their life, and clean
up all internal strong references associated with them.

This commit also fixes mockito#1532 and fixes mockito#1533.
ttanxu added a commit to ttanxu/mockito that referenced this issue Feb 28, 2019
Due to the introduction of map from weak reference from mock instance to
its invocation handler, Mockito became vunerable to memory leaks as
there are multiple situations where Mockito could unintentionally hold
strong references to mock instances in the map record. The strong
references could be through spiedInstance for spies, and arguments used
to faciliate method stubbing.

Mockito could never know if the arguments passed in for method stubbing
are also strongly referenced somewhere else or not, so Mockito needs to
save a strong reference to these arguments to avoid premature GC.
Therefore to solve cyclic strong references through arguments Mockito
needs to explicitly know when mocks are out of their life, and clean
up all internal strong references associated with them.

This commit also fixes mockito#1532 and fixes mockito#1533.
ttanxu added a commit to ttanxu/mockito that referenced this issue Mar 1, 2019
Due to the introduction of map from weak reference from mock instance to
its invocation handler, Mockito became vunerable to memory leaks as
there are multiple situations where Mockito could unintentionally hold
strong references to mock instances in the map record. The strong
references could be through spiedInstance for spies, and arguments used
to faciliate method stubbing.

Mockito could never know if the arguments passed in for method stubbing
are also strongly referenced somewhere else or not, so Mockito needs to
save a strong reference to these arguments to avoid premature GC.
Therefore to solve cyclic strong references through arguments Mockito
needs to explicitly know when mocks are out of their life, and clean
up all internal strong references associated with them.

This commit also fixes mockito#1532 and fixes mockito#1533.
@jemshit
Copy link

jemshit commented Mar 15, 2019

Wow, Eclipse

@arturdryomov
Copy link
Author

Just checked Mockito 2.27.0. Using the new Mockito.framework().clearInlineMocks() call does the trick and helps to avoid OOM. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants