Connecting over gRPC from Backend with cerbos-java-sdk to Cerbos fails in AKS

[bbodensieck]

Hey folks,

we are having the following

Setup

• Java Spring Boot Backend with cerbos-sdk-java:0.7.0 connecting to port 3593
• Cerbos (storage on Mysql, Admin Api is activated)
• A Cerbos job, which puts policies into Cerbos via ctl.

Local setup ✅

• Docker compose ✅
• Everything works like a charm ✅

Cloud setup

• AKS (Kubernetes on Azure)
• Helm Charts (Cerbos is deployed as a Service)
• Cerbos Job can put policies intro Cerbos ✅
• Backend can not communicate with Cerbos over gRPC ❌

The communication between our backend and cerbos is just internal inside the cluster.

Do you have any idea which could help us?

We are getting the following exception:

RPC exception [Status{code=INTERNAL, description=http2 exception, cause=io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception: First received frame was not SETTINGS. Hex dump for first 5 bytes: 485454502f at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:109) at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.verifyFirstFrameIsSettings(Http2ConnectionHandler.java:353) at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:247) at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:453) at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800) at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499) at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:833) }]

Hoping for help. Have a great day,
Björn

[charithe]

It looks like your backend is unable to establish a HTTP/2 connection to Cerbos. Is it connecting to Cerbos through a load balancer or a proxy of some sort? Some of them need to be explicitly configured to support HTTP/2.

[charithe]

Hmm... I am baffled. Could you try specifying the target without the dns:// prefix? So, something like cerbos-svc.my-namespace.svc.cluster.local:3593.

[bbodensieck]

A small working Kubernetes example with a java 17 backend, which uses cerbos-java-sdk and is able to connect to cerbos via CerbosBlockingClient and CerbosBlockingAdminClient would be great. Maybe you have something from your testing environment? ✨

[charithe]

We don't have a Kubernetes example using Java but I just tried it out. The gRPC library relies on Java SPI to register name resolvers and client-side load balancers. The default are defined in the io.grpc:grpc-core library. So, as a first step, I suggest adding that as a dependency to your Java project (implementation("io.grpc:grpc-core:1.56.1") in Gradle).

Sometimes, depending on how the Java jar is produced, the SPI registrations defined in META-INF/services get stripped out or overwritten. In that case, you could explicitly register them in code before the Cerbos client is initialised. Here's an example of doing that:

import dev.cerbos.api.v1.engine.Engine;
import dev.cerbos.sdk.builders.Principal;
import dev.cerbos.sdk.builders.Resource;
import io.grpc.LoadBalancerRegistry;
import io.grpc.NameResolverRegistry;
import io.grpc.internal.PickFirstLoadBalancerProvider;

public class Cerbos {
    public static void main(String[] args) throws CerbosClientBuilder.InvalidClientConfigurationException {
        LoadBalancerRegistry.getDefaultRegistry().register(new PickFirstLoadBalancerProvider());
        NameResolverRegistry.getDefaultRegistry().register(new io.grpc.internal.DnsNameResolverProvider());
        CerbosBlockingClient client = new CerbosClientBuilder("dns:///cerbos.infra.svc.cluster.local:3593").withInsecure().buildBlockingClient();
        ...
    }
}

[bbodensieck]

Thank you so much for helping us investigating.

In the end there was an issue with the port which was set in the deployment into the environment variable and over the application.yml. At some place it got lost.

Now it works properly without explicitly using io.grpc:grpc-core and with just cerbos-svc:3593 as targetUrl.

So everything is fine now.

Thank you again for providing us great help. We really appreciate.
Have a great day,
Björn