Invalid or unsupported protocol version (4); supported versions are (3/v3, 4/v4, 5/v5-beta)

[txynidakis]

Hi
We were running helm chart v0.71.3, and we migrated to karpenter as our worker nodes in EKS. After the node migration, both the jaeger collector and query pods failed to start.

The query pods seem to be failing because there are no collector pods in service.

The collector container logs contain the following entries

2024/05/01 05:42:43 gocql: unable to dial control conn 10.207.181.98:9042: Invalid or unsupported protocol version (4); supported versions are (3/v3, 4/v4, 5/v5-beta)
{"level":"fatal","ts":1714542163.2535214,"caller":"collector/main.go:82","msg":"Failed to init storage factory","error":"gocql: unable to create session: unable to connect to initial hosts: Invalid or unsupported protocol version (4); supported versions are (3/v3, 4/v4, 5/v5-beta)","stacktrace":"main.main.func1\n\tgithub.com/jaegertracing/jaeger/cmd/collector/main.go:82\ngithub.com/spf13/cobra.(*Command).execute\n\tgithub.com/spf13/cobra@v1.8.0/command.go:983\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/cobra@v1.8.0/command.go:1115\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/cobra@v1.8.0/command.go:1039\nmain.main\n\tgithub.com/jaegertracing/jaeger/cmd/collector/main.go:155\nruntime.main\n\truntime/proc.go:267"}

We updated to Helm chart version 3.0.4, but the same error appeared.

The Jaeger-Cassandra (stateful set) pods (3) are in a running state.

Can anyone shed some light on the issue, or any other diagnostic tips?