Using httplib OutOfMemoryError

[fit2-zhao]

Using httplib in JMeter will cause Metaspace to continue to rise until java.lang.OutOfMemoryError, how to solve it, the current version is 2.7.3

[fit2-zhao]

2.7.0 version is normal

[thetric]

Could you provide a minimal example to reproduce the issue? It is hard to pinpoint the problem without it

[fit2-zhao]

Thank you very much for taking the time to reply to my question. This is my JMeter script. It will be OOM when executed in JMeter 5.x。
OutOfMemoryError-Test.TXT

[fit2-zhao]

java.lang.OutOfMemoryError: Metaspace
Jython internals might be in a bad state now that can cause deadlocks later on.
See http://bugs.jython.org/issue2536 for details.
@thetric Is there a plan to fix this?

The same script is executed normally in version 2.7.0 .

[vlsi]

The root cause is that PyScriptEngineFactory.getScriptEngine() returns a new script engine every time (see

jython/src/org/python/jsr223/PyScriptEngineFactory.java

Line 85 in c0a5d43

return new PyScriptEngine(this);

), and each engine spawns Netty threads.

Unfortunately, JMeter does not close ScriptEngine after each use, so Jython has no opportunity to close the world and terminate runaway Netty threads.

Unfortunately, JMeter does not provide "notion of script engine context" yet. In other words, there's no way to tell if JSR223 Sampler should create new context on every invocation or if it should reset the interpreter on each execution.

So the solution could be:

1. Detect and cleanup runaway interpreters at jython side (e.g. automatically close interpreters if the client does not call .close())
2. Deal with JSR223 scope somehow in JMeter. For instance, make sure JMeter creates and caches interpreter per thread or something like that. However, if we initialize ScriptEngine per JMeterThread, it would result in the same "lots of duplicated python classes" memory leak, so we would probably need something like "pooling interpreters" (e.g. creating one or two interpreters and reusing them across threads)

[jeff5]

@vlsi: very helpful analysis, thanks.

The handling of multiple interpreters in Jython 2 does not quite make sense to me, but then in CPython 2 and many 3.x-es it had to be hedged with warnings. It is the best that clever people could do at the time with some subsequent shoring-up.

JSR 223 then makes stretching demands this (possibly flawed) foundation.

CPython 3 has recently re-thought the implementation of multiple interpreters, which gives us a guide in the hoped-for Jython 3.

[vlsi]

Do you mean the current Jython does not support multiple concurrent interpreters?

[jeff5]

I mean that I find the way it does so difficult to follow/use. Evidently it has been used successfully. I would hesitate to change it.

I have followed the CPython re-think with interest (PEP-684 and others).

[fit2-zhao]

Will this problem be fixed in the future? @jeff5

[fit2-zhao]

At present, this problem only occurs when importing httplib.

[jeff5]

Will this problem be fixed in the future? @jeff5

It depends what "the problem" is (and which future). The problem might be:

• Jython creates a new interpreter with its own sys and other modules every time you ask for a ScriptEngine. Probably not in Jython 2. That has been the case for a long time (ef262f4) and I assume with good reason. It may not be exactly the same, but the separate interpreters in CPython have the same separation. There may be a way to vary this in a Jython 3.
• Netty behaves poorly in Jython when stressed (e.g. in our tests), perhaps because we are using it incorrectly. I have looked for fixes in Jython 2 but do not have the time or knowledge. My preference is to start again in Jython 3 without Netty.
• Jython keeps an interpreter around even when you have dropped your reference to it. It depends on the root cause. If we are keeping a reference internally that could have been a weak one, then possibly. If it is because a real thread (e.g. Netty's event loop) is still running then previous answer.

[fit2-zhao]

Thank you very much for your answer.