ByteBuf.release() was not called before it's garbage-collected #5978

Umang-Goel · 2019-07-11T12:05:06Z

Please answer these questions before submitting your issue.

What version of gRPC are you using?

What did you expect to see?

No leak

Actual behavior

ERROR 2019-07-11T03:16:07,856 ResourceLeakDetector:171 [pool-2-thread-39] LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
#1:
        Hint: 'NettyServerHandler#0' will handle the message from this point.
        io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1407)
        io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1177)
        io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1221)
        io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
        io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
        io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
        io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
        io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:647)
        io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:582)
        io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499)
        io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:461)
        io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.lang.Thread.run(Thread.java:812)
#2:
        io.netty.buffer.AdvancedLeakAwareByteBuf.nioBufferCount(AdvancedLeakAwareByteBuf.java:706)
        io.netty.handler.ssl.SslHandler.toByteBuffer(SslHandler.java:1417)
        io.netty.handler.ssl.SslHandler.access$300(SslHandler.java:165)
        io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:209)
        io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1275)
        io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1177)
        io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1221)
        io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
        io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
        io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
        io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
        io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:647)
        io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:582)
        io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499)
        io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:461)
        io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.lang.Thread.run(Thread.java:812)
#3:
        io.netty.buffer.AdvancedLeakAwareByteBuf.internalNioBuffer(AdvancedLeakAwareByteBuf.java:736)
        io.netty.handler.ssl.SslHandler.toByteBuffer(SslHandler.java:1417)
        io.netty.handler.ssl.SslHandler.access$300(SslHandler.java:165)
        io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:209)
        io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1275)
        io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1177)
        io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1221)
        io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
        io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
        io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
        io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
        io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:647)
        io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:582)
        io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499)
        io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:461)
        io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.lang.Thread.run(Thread.java:812)
Created at:
        io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:331)
        io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:185)
        io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:176)
        io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:1887)
        io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1270)
        io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1177)
        io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1221)
        io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
        io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
        io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
        io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
        io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:647)
        io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:582)
        io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499)
        io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:461)
        io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.lang.Thread.run(Thread.java:812)
: 1 leak records were discarded because they were duplicates
: 1 leak records were discarded because the leak record count is targeted to 4. Use system property io.netty.leakDetection.targetRecords to increase the limit.
ERROR 2019-07-11T03:16:07,874 ResourceLeakDetector:171 [grpc-default-worker-ELG-3-12] LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
#1:
        Hint: 'NettyServerHandler#0' will handle the message from this point.
        io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1407)
        io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1177)
        io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1221)
        io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
        io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
        io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
        io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
        io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:647)
        io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:582)
        io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499)
        io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:461)
        io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.lang.Thread.run(Thread.java:812)
#2:
        io.netty.buffer.AdvancedLeakAwareByteBuf.internalNioBuffer(AdvancedLeakAwareByteBuf.java:736)
        io.netty.handler.ssl.SslHandler.toByteBuffer(SslHandler.java:1417)
        io.netty.handler.ssl.SslHandler.access$300(SslHandler.java:165)
        io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:209)
        io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1275)
        io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1177)
        io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1221)
        io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
        io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
        io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
        io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
        io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:647)
        io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:582)
        io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499)
        io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:461)
        io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.lang.Thread.run(Thread.java:812)
#3:
        io.netty.buffer.AdvancedLeakAwareByteBuf.nioBufferCount(AdvancedLeakAwareByteBuf.java:706)
        io.netty.handler.ssl.SslHandler.toByteBuffer(SslHandler.java:1417)
        io.netty.handler.ssl.SslHandler.access$300(SslHandler.java:165)
        io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:209)
        io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1275)
        io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1177)
        io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1221)
        io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
        io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
        io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
        io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
        io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:647)
        io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:582)
        io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499)
        io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:461)
        io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.lang.Thread.run(Thread.java:812)
Created at:
        io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:331)
        io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:185)
        io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:176)
        io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:1887)
        io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1270)
        io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1177)
        io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1221)
        io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
        io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
        io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
        io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
        io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:647)
        io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:582)
        io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499)
        io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:461)
        io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.lang.Thread.run(Thread.java:812)

Flags used to launch jvm:
-Xmx256m
-XX:MaxDirectMemorySize=256m

Also getting below logs:

Jul 11, 2019 3:31:10 AM io.grpc.netty.NettyServerTransport notifyTerminated
INFO: Transport failed
io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 251658527, max: 268435456)
        at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:640)
        at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:594)
        at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:764)
        at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:740)
        at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:244)
        at io.netty.buffer.PoolArena.allocate(PoolArena.java:226)
        at io.netty.buffer.PoolArena.allocate(PoolArena.java:146)
        at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:324)
        at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:185)
        at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:176)
        at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:137)
        at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:147)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:647)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:582)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:461)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:812)

JVM version (e.g. `java -version`)

Java(TM) SE Runtime Environment (build 8.0.5.26 - pap6480sr5fp26-20181115_03(SR5 FP26)) IBM J9 VM (build 2.9, JRE 1.8.0 AIX ppc64-64-Bit Compressed References 20181106_401576 (JIT enabled, AOT enabled) OpenJ9 - fde1d6f OMR - d8c3617 IBM - 5c4a9f0) JCL - 20181022_01 based on Oracle jdk8u191-b26

OS version (e.g. `uname -a`)

-bash-4.4# uname -a
AIX lpar3 2 7 00FB51394C00

I got the above log by setting -Dio.netty.leakDetection.level=paranoid.
From the log it seems like NettyServerHandler does not correctly call release() on the ReferenceCounted objects it receives. But, I'm not explicitly using NettyServerHandler anywhere in the codebase. Though I'm using grpc and following are the classes which are being used:

io.grpc.Server
io.grpc.ServerBuilder
io.grpc.stub.StreamObserver

Do you have any idea which of these might be using the NettyServerHandler?

The text was updated successfully, but these errors were encountered:

ejona86 · 2019-07-12T16:17:49Z

The ByteBufs are passed to the application thread for processing. Are you using ServerInterceptors.useInputStreamMessages() or making your own MethodDescriptor.Marshaller? Are you specifying your own executor to ServerBuilder.executor()?

Umang-Goel · 2019-07-15T08:44:36Z

@ejona86
Yes, I am specifying my own executor to ServerBuilder.executor().
Please find the code below:

public class AgentMain {

  /** The default port on which grpc listens on. */
  private static int FLAGS_GRPC_SERVER_PORT;

  /** The number of threads to use for serving the requests. */
  private static int FLAGS_GRPC_THREADPOOL_SIZE;

  /** The gRPC server_. */
  private final Server server_;

  AgentMain() {
    ServerBuilder serverBuilder =
        ServerBuilder.forPort(FLAGS_GRPC_SERVER_PORT)
            .executor(Executors.newFixedThreadPool(FLAGS_GRPC_THREADPOOL_SIZE))
            .addService(new Service1());
    server_ = serverBuilder.build();
  }

  /** Start serving requests. */
  public void start() throws IOException {
    server_.start();
    Runtime.getRuntime()
        .addShutdownHook(
            new Thread() {
              @Override
              public void run() {
                AgentMain.this.stop();
              }
            });
  }

  public void stop() {
    if (server_ != null) {
      server_.shutdown();
    }
  }

  public void blockUntilShutdown() throws InterruptedException {
    if (server_ != null) {
      server_.awaitTermination();
    }
  }

  public static void main(String[] args) {
    try {
      FLAGS_GRPC_SERVER_PORT = 12345;
      FLAGS_GRPC_THREADPOOL_SIZE = 128;
      AgentMain agent = new AgentMain();
      agent.start();
      agent.blockUntilShutdown();
    } catch (Exception e) {
      // Log the exception
    }
  }
}

Umang-Goel · 2019-07-17T06:51:34Z

Update : I removed the function call ServerBuilder.executor(). Now the server command looks like this :

ServerBuilder serverBuilder =
        ServerBuilder.forPort(FLAGS_GRPC_SERVER_PORT)
            .addService(new Service1());
    server_ = serverBuilder.build();

But still I'm getting the same error.

carl-mastrangelo · 2019-07-17T08:29:21Z

@Umang-Goel are you sure that's the right code? It's missing semicolons after the server builder.

Umang-Goel · 2019-07-17T08:43:24Z

@carl-mastrangelo yes, it's correct. I made a typo while copying it here.

ejona86 · 2019-07-17T16:24:36Z

.executor(Executors.newFixedThreadPool(FLAGS_GRPC_THREADPOOL_SIZE)) is safe, because you are relying on the finalizer to shut down the thread pool (so there's no chance of Runnables being lost and never executed).

If you were using blocking stub I could have more ideas of what is going wrong, but you're using async stub. And it doesn't appear you are using interceptors. So I don't really know how this could be happening.

@Umang-Goel, do you have any idea on what type of RPC triggers this? I'd assume it isn't happening every RPC. The most likely places something like this could happen is in error-handling code paths.

ejona86 · 2019-07-30T17:53:57Z

Unclear how to diagnose this more without a reproduction case.

But we could add a touch() before we hand off to the MessageDeframer so that we know a bit more of where this buffer was lost.

amandeepgautam · 2019-08-21T20:50:38Z

@ejona86 @creamsoup any updates on this? do you need anything from us?

creamsoup · 2019-08-21T21:24:32Z

@amandeepgautam thanks for the ping.
we added touch() before the MessageDeframer as @ejona86 suggested. but, this didn't make the 1.23.0 branch cut. Unless you can get the leak detector log using 1.24.0-SNAPSHOT (or patching the commit), i am afraid to say that there's not much we can do without the log or repro.

amandeepgautam · 2019-08-22T23:40:37Z

@creamsoup What do you mean by "leak detector log"? Can you help us with the steps to collect these?

creamsoup · 2019-08-22T23:46:59Z

the first log of this issue has the leak detector log. also see this link from netty.

@amandeepgautam if you are also seeing similar issue, you can turn on the leak detection level even higher by providing jvm flag -Dio.netty.leakDetection.level=$HIGHER_LEVEL which should increase the chance of leak detection.

hongliang5623 · 2021-02-23T07:08:39Z

@ejona86 @creamsoup i have the same problem, the server memory usage continues to grow to the maximum available, my grpc-java version is 1.23, and my java version is 1.8.0_202, Is there any update?

LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
#1:
        io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.expandCumulation(ByteToMessageDecoder.java:534)
        io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder$1.cumulate(ByteToMessageDecoder.java:92)
        io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:281)
        io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
        io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421)
        io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
        io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:794)
        io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:424)
        io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:326)
        io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
        io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.lang.Thread.run(Thread.java:748)

#2:
		io.grpc.netty.shaded.io.grpc.netty.NettyClientStream$TransportState.transportDataReceived(NettyClientStream.java:379)
		io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler.onDataRead(NettyClientHandler.java:376)
		io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler.access$1100(NettyClientHandler.java:89)
		io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler$FrameListener.onDataRead(NettyClientHandler.java:842)
		io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(DefaultHttp2ConnectionDecoder.java:292)
		io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onDataRead(Http2InboundFrameLogger.java:48)
		io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readDataFrame(DefaultHttp2FrameReader.java:422)
		io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:251)
		io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:160)
		io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41)
		io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:174)
		io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:378)
		io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:438)
		io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:505)
		io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444)
		io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283)
		io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
		io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
		io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
		io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421)
		io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
		io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
		io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
		io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:794)
		io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:424)
		io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:326)
		io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
		io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
		io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
		java.lang.Thread.run(Thread.java:748)
Created at:
        io.grpc.netty.shaded.io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:349)
        io.grpc.netty.shaded.io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
        io.grpc.netty.shaded.io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
        io.grpc.netty.shaded.io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:115)
        io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.expandCumulation(ByteToMessageDecoder.java:532)
        io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder$1.cumulate(ByteToMessageDecoder.java:92)
        io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:281)
        io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
        io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421)
        io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
        io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:794)
        io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:424)
        io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:326)
        io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
        io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.lang.Thread.run(Thread.java:748)

voidzcy · 2021-02-23T21:21:59Z

1.23 is quite old. Please try newer versions. #7105 found some issue that can cause the leak, but we mostly considered it to be #6328 (which had been reverted) introduced in 1.27. The potential message leaking issue should have been improved by #7187 and #7798 (not merged yet). While not 100% sure if your case is caused by related/similar reason, it worths to upgrade (>1.33) and see if the problem persists.

hongliang5623 · 2021-02-25T03:50:54Z

@voidzcy thank you for your reply, I found the server memory usage is stable at night , and the speed of memory leak is strongly related to the amount of requests, This is very puzzling，i will try the upgrade and watch，thanks again~

iamtomkeen · 2023-02-10T14:20:08Z

Hello @voidzcy,
I'm facing this issue wih 1.47. Would you have any suggestion on how to catch it?

ejona86 · 2023-02-10T16:07:43Z

I'm closing this because it very well could have been fixed and we made improvements that will change the error message to provide us more information. If you see something resembling this, it is best to create a new issue and provide the dumped stack traces for us to identify the specific cause. It would be hard for users to determine if they are seeing the same issue.

I will note that 1.49 fixed one leak, and the release-in-progress-1.53 fixes another. Both of those are caused by races in the retry code when calls are cancelled; using managedChannelBuilder.disableRetry() works around the bugs.

dapengzhang0 assigned carl-mastrangelo Jul 12, 2019

Umang-Goel mentioned this issue Jul 12, 2019

LEAK: ByteBuf.release() was not called before it's garbage-collected netty/netty#9350

Closed

ejona86 assigned creamsoup and unassigned carl-mastrangelo Jul 30, 2019

creamsoup added this to the Next milestone Jul 30, 2019

creamsoup removed their assignment Jul 8, 2020

ejona86 closed this as completed Feb 10, 2023

ejona86 removed this from the Next milestone Feb 10, 2023

github-actions bot locked as resolved and limited conversation to collaborators May 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ByteBuf.release() was not called before it's garbage-collected #5978

ByteBuf.release() was not called before it's garbage-collected #5978

Umang-Goel commented Jul 11, 2019

ejona86 commented Jul 12, 2019

Umang-Goel commented Jul 15, 2019 •

edited

Umang-Goel commented Jul 17, 2019 •

edited

carl-mastrangelo commented Jul 17, 2019

Umang-Goel commented Jul 17, 2019

ejona86 commented Jul 17, 2019

ejona86 commented Jul 30, 2019

amandeepgautam commented Aug 21, 2019

creamsoup commented Aug 21, 2019

amandeepgautam commented Aug 22, 2019 •

edited

creamsoup commented Aug 22, 2019

hongliang5623 commented Feb 23, 2021

voidzcy commented Feb 23, 2021 •

edited

hongliang5623 commented Feb 25, 2021

iamtomkeen commented Feb 10, 2023

ejona86 commented Feb 10, 2023

ByteBuf.release() was not called before it's garbage-collected #5978

ByteBuf.release() was not called before it's garbage-collected #5978

Comments

Umang-Goel commented Jul 11, 2019

What version of gRPC are you using?

What did you expect to see?

Actual behavior

JVM version (e.g. java -version)

OS version (e.g. uname -a)

ejona86 commented Jul 12, 2019

Umang-Goel commented Jul 15, 2019 • edited

Umang-Goel commented Jul 17, 2019 • edited

carl-mastrangelo commented Jul 17, 2019

Umang-Goel commented Jul 17, 2019

ejona86 commented Jul 17, 2019

ejona86 commented Jul 30, 2019

amandeepgautam commented Aug 21, 2019

creamsoup commented Aug 21, 2019

amandeepgautam commented Aug 22, 2019 • edited

creamsoup commented Aug 22, 2019

hongliang5623 commented Feb 23, 2021

voidzcy commented Feb 23, 2021 • edited

hongliang5623 commented Feb 25, 2021

iamtomkeen commented Feb 10, 2023

ejona86 commented Feb 10, 2023

JVM version (e.g. `java -version`)

OS version (e.g. `uname -a`)

Umang-Goel commented Jul 15, 2019 •

edited

Umang-Goel commented Jul 17, 2019 •

edited

amandeepgautam commented Aug 22, 2019 •

edited

voidzcy commented Feb 23, 2021 •

edited