Enabling TLS for grpc causes the readiness probe to fail

[kdubb]

Adding the -tls flag to the probe command along with and adding 0.0.0.0 as an IP SAN to the certificate via kustomization fixed the issue.

[kdubb]

Once the chart was rendering healthy pods I wanted to tweak the readiness probe to be able to remove the required 0.0.0.0 from the generated cert. As of yet I cannot get that to work.

Context, the release I am working with is named openfga and openfga is one of the DNS SANS in the TLS cert; I have left the 0.0.0.0 IP SAN in as well.

The following command, run from the healthy container, succeeds:

/bin/grpc_health_probe -addr=localhost:8081 -tls -tls-server-name=openfga

While this command does not:

/bin/grpc_health_probe -addr=localhost:8081 -tls

This tells me that the hardcoded alternate server name (provided via -tls-server-name) is working for grpc_health_probe.

If I remove the 0.0.0.0 from the certificate and use the successful command above for my readiness probe, the pod fails to start with context deadline exceeded. So there must be something with the TLS server and the 0.0.0.0 address but I do not know what it is.

[kdubb]

In case anybody is wondering, the reason I am not using openfga in the -addr option is that is the name of the generated service; obviously not usable for the readiness probe for an individual pod.

[kdubb]

After a bit of digging. This 0.0.0.0 issue has to do with a bug in the server itself (openfga/openfga#640).

This issue remains valid as the tls options need to be applied to the grpc_health_check.

When TLS is enabled, the chart needs to add a grpc.tls.serverName option to pass to grpc_health_check as -tls-server-name or it needs to always add -tls-no-verify.