xDS resolver error: received message larger than max

[technicianted]

For larger number of services (~700 in our case), gRPC xDS client fails to read data due to max receive message size exceeded.

A workaround is to increase the limit and is confirmed to fix the problem:

diff --git a/xds/internal/xdsclient/controller/controller.go b/xds/internal/xdsclient/controller/controller.go
index 4b07dc8d..1c679b56 100644
--- a/xds/internal/xdsclient/controller/controller.go
+++ b/xds/internal/xdsclient/controller/controller.go
@@ -118,6 +118,7 @@ func New(config *bootstrap.ServerConfig, updateHandler pubsub.UpdateHandler, val
                        Time:    5 * time.Minute,
                        Timeout: 20 * time.Second,
                }),
+               grpc.WithDefaultCallOptions(grpc.MaxCallRecvMsgSize(16 * 1024 * 1024)),
        }
 
        if boff == nil {

There should be a way to configure this and potentially other xDS client call/channel options, perhaps via env variables.

What version of gRPC are you using?

1.50.1

What version of Go are you using (go version)?

1.18.5

What operating system (Linux, Windows, …) and version?

Linux

What did you do?

Using xDS resolver with Istio side car for xds:/// endpoint.

What did you expect to see?

Call works.

What did you see instead?

name resolver error: xds: error received from xDS stream: xds: stream.Recv() failed: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (4373177 vs. 4194304)

[easwars]

Thank you for filing this issue. We are still having discussions about what would be the best way forward for this. Will update this thread shortly.

[easwars]

We are going to be sending out a gRFC for this very soon. Hence, marking this issue blocked on it.

[easwars]

The gRFC is here: grpc/proposal#384