maxInboundMessageSize is not applied when app code has newer gRPC version than used in library

[marx-freedom]

What version of gRPC-Java are you using?

Library compiled with gRPC 1.26.0 (cannot be recompiled on demand)
Application must use a newer version (>= 1.34), gRPC 1.39.0

What is your environment?

Linux, OpenJDK 15

What did you expect to see?

Custom maxInboundMessageSize value (!= 4 MiB) is applied and works correctly

What did you see instead?

Status{code=CLIENT_RESOURCE_EXHAUSTED, issues=[gRPC error: (RESOURCE_EXHAUSTED) gRPC message exceeds maximum size 4194304: 14995791 (S_ERROR)]}

Steps to reproduce the bug

1. Compile library code with gRPC 1.26.0

import io.grpc.ManagedChannelBuilder;
import io.grpc.netty.NettyChannelBuilder;

public class Lib {
  private final NettyChannelBuilder ncb;

  public Lib() {
    ncb = NettyChannelBuilder
        .forTarget("localhost:100")
        .maxInboundMessageSize(100);
  }

  public NettyChannelBuilder getBuilder() {
    return ncb;
  }
}

2. Compile application, using the compiled library, with gRPC 1.39.0

import java.lang.reflect.Field;
import io.grpc.netty.NettyChannelBuilder;
import io.grpc.internal.AbstractManagedChannelImplBuilder;

public class Client {

  public static void main(String[] args) throws Exception {
    NettyChannelBuilder ncb = new Lib().getBuilder();
    // reflection is used to create a minimal example showing the problem - 
    // no need to create a real channel, start the server, etc.
    Field f = ncb.getClass().getDeclaredField("maxInboundMessageSize");
    f.setAccessible(true);
    System.out.println(f.get(ncb));
  }
}

3. Run compiled Client with gRPC 1.39.0 in the classpath.
   The program will print 4194304 but value 100 expected.

It seems this behavior was caused by fixing issue #7552

[ericgribkoff]

I don't fully understand the mixing of old-and-new versions of gRPC you are using when you describe reproducing this case, but it does indeed seem related to the ABI issue introduced at 1.33 and subsequently fixed (#7552, as you point out).

@sergiitk any thoughts here?

[marx-freedom]

In the example, I am modeling a real situation: the library compiled with the old version is used as a binary dependency (and cannot be recompiled in nearest future).

Fortunately, I've found a workaround (the library allows you to customize ChannelBuilder). I decided to report this because ABI issue was fixed in #7552 and someone may suddenly encounter the same issue.

[sergiitk]

@marx-freedom Thanks for documenting this case. That's an interesting one.

As I understand, this might be not #7552 specifically, but rather it's a consequence of one of many PRs that make up the larger refactoring #7211. This refactoring broke the ABI, and #7552 rolled back some parts of the refactoring, and rolled forward other.

Looks like the issue is caused in #7359 by pushing down maxInboundMessageSize field to NettyChannelBuilder as a private one. Or by overriding its setter maxInboundMessageSize(int).

It's possible this is another case of ABI breakage. cc @ejona86
I'll use your example to reproduce the issue. I want to step through this with the debugger to understand the problem better, and confirm if there's something to be done on our side.

[ejona86]

This is caused by #7834.

$ javap netty/build/classes/java/main/io/grpc/netty/NettyChannelBuilder.class  | grep maxInboundMessageSize
  public io.grpc.netty.NettyChannelBuilder maxInboundMessageSize(int);
  public io.grpc.ManagedChannelBuilder maxInboundMessageSize(int);

The problem is when compiling NettyChannelBuilder the generic T doesn't extend AbstractManagedChannelImplBuilder so javac doesn't create the additional variation of the method that returns AbstractManagedChannelImplBuilder. However AbstractManagedChannelImplBuilder itself does have the method, and it just forwards, which is then thrown away by the default implementation in ManagedChannelBuilder. This isn't a problem for the (most?) other methods because they are actually implemented by ManagedChannelImplBuilder, but in this case we moved the responsibility of which class handles the method which exposes a gap.

A solution is not immediately clear. I'm toying with ideas like having AbstractManagedChannelImplBuilder.maxInboundMessageSize() throw; at least then this won't be a silent issue. We should check to find any other methods that may be similarly impacted.

[ejona86]

BTW, @marx-freedom, good job tracking the issue down to the clear reproduction steps. I'm sure it was very confusing.

[sergiitk]

Hey @marx-freedom! Good news, I tracked down exact root causes, and fixed the issue. After the fix is released, and you update your application code, maxInboundMessageSize(int) will be working correctly.