GRPCRoute conformance

[mlavacca]

Is there an existing issue for this?

• I have searched the existing issues

Problem Statement

Gateway API v1.1 will be released soon with a new set of GRPC conformance tests, and the graduation of the GRPCRoute API. Since we are implementing the GRPCRoute, we want to run the new GRPCRoute tests and claim conformance with it.

Acceptance Criteria

• The GRPCRoute conformance tests are successfully run.
• The default behavior changes only affect feature-gated alpha capability
• Documentation explicitly covers both GRPC and GRPCS backend cases.

[tao12345666333]

Let me pick up this one.

[tao12345666333]

Update:

We encountered errors in some conformance test cases.

This happens because when the KIC translator processes the GRPCRoute resource, it sets the default protocol to grpcs instead of grpc.

kubernetes-ingress-controller/internal/dataplane/translator/translate_grpcroute.go

Lines 56 to 59 in 5701912

	// Create a service and attach the routes to it. Protocol for Service can be set via K8s object annotation
	// "konghq.com/protocol", by default use "grpcs" to not break existing behavior when annotation is not specified.
	service, err := generateKongServiceFromBackendRefWithRuleNumber(
	t.logger, t.storer, result, grpcroute, ruleNumber, "grpcs", grpcBackendRefsToBackendRefs(rule.BackendRefs)...,

More background context here:

• GRPCRoute: Not supported by HTTP Listener. #4273
• fix: restore defaulting to protocol "grpcs" for gRPC Service #5283

When we implemented this feature, the Kong Gateway version used was 3.5, which is based on OpenResty 1.21.4.2. Since version 3.6, it is based on 1.25.3.1 which added support for "Allow h2c and HTTP/1.1 support on the same listening socket"

We have set this default value grpcs in KIC#5283 because we don't want to break users existing configurations.

• However, support for GRPCRoute is configured through GatewayAlpha and is not enabled by default.
• As Gateway API's GRPCRoute is about to become generally available, I think we should consider modifying this behavior. We will implement the GRPCRoute v1 API, and it will be enabled like other GWAPI resources.

Why we need to change the default GRPCRoute's protocol

• pass the GWAPI's conformance test
• I have checked other GWAPI implementations. All of them's default implementation is grpc instead of grpcs.

[mflendrich]

Decision reached with @tao12345666333 :

• Since the GRPCRoute implementation is alpha and feature-gated, we will change the default behavior in a minor version of KIC
• The changelog entry will explain how to migrate given the new default
• The documentation will explicitly cover both the "GRPC non-TLS backend" and the "GRPCS" cases.

[rainest]

What's the actual conformance test that fails? Is it actually failing because the upstream traffic is sent over grpcs instead of grpc. or just because the gate is disabled?

If conformance is deploying an application that only supports plaintext, we should probably actually try to change that upstream, since it should be controlled by the Service, not any of the GWAPI resources, and shouldn't be a test requirement as such.

None of

https://gateway-api.sigs.k8s.io/guides/grpc-routing/
https://gateway-api.sigs.k8s.io/concepts/api-overview/#grpcroute
https://gateway-api.sigs.k8s.io/api-types/grpcroute/

appear to mention any default, and https://gateway-api.sigs.k8s.io/geps/gep-1016/#http2-cleartext mentions that this is more for testing, so it probably shouldn't be a default.

AFAICT the actual tests only care that we reach the correct backend, and don't see which protocol we use to do so. The Deployments created by https://github.com/kubernetes-sigs/gateway-api/blob/main/conformance/base/manifests.yaml do indeed not support TLS based on basic testing, but we shouldn't take that as a de facto part of the standard, it's just a shortcoming of the test environment.

Those Services arguably should indicate their protocol explicitly, though there's apparently no standard or de facto standard value for gRPC in appProtocol.

https://gateway-api.sigs.k8s.io/api-types/backendtlspolicy/ is another option, but I don't think we have any support for it yet (and the test manifests don't currently include it).

@shaneutt do you know anything about this? The tests apparently impose a hidden requirement that's arguably at odds with typical production gRPC expectations, where using encrypted HTTP/2 as the underlying transport is the norm.

[shaneutt]

I'm unavailable and wont be able to dig in for the next few days. I did however share this thread with other Gateway API folks and we're agreed to hold on releasing v1.1 until we've got things sorted out. @tao12345666333, @mlavacca: if you're working on this the next couple days and have any updates, let Rob and Richard know any updates (on Kubernetes slack) as they're aware of the situation.

[tao12345666333]

The reason why the conformance test fails is because the test case only uses plaintext, but since KIC sets the protocol to grpcs, it will fail.

I have tested that it works correctly if set to grpc.

Thanks @rainest and @shaneutt. I will reach out to the GWAPI team.

[tao12345666333]

Interim conclusion:

• GWAPI will bring test cases for grpcs, and the current test case is clearly defined as h2c protocol through app-protocol. Therefore, KIC also needs to support the h2c protocol.

I will complete this goal through the following steps to avoid generating a huge PR and facilitate review.

• allow installation using a specified GatewayAPI commit test: allow installation using a specified GatewayAPI commit #5808

• Upgrade the GWAPI version to the commit before GRPCRoute GA, so that KIC can better complete the adaptation of GWAPI. For example, experimental conformance testing has become a standard. test(conformance): upgrade gwapi version #5863

• Bump GWAPI to v1.1 and introduce GRPCRoute v1 support feat: bump GWAPI v1.1 and support for GRPCRoute v1 #5918

  • Bump client-go and controller-runtime chore(dep): group update k8s.io to v0.30.0 and controller-runtime to v0.18.2 #5935

• Make KIC support h2c protocol to pass the conformance test conformance: add support for GRPCRoute #5776

[tao12345666333]

since #5776 hasn't been merged, I will reopen this issue.