[Bug]: Adding a serviceAccount annotation in KafkaConnect spec makes Cluster Operator Stuck on Reconciliation of Strimzipodset

[yordanovsstoyan]

Bug Description

Adding an annotation to the service account of Kafka Connect creates only the new annotated service account (kafka-connect pod does not get created), while cluster operator times out trying to reconcile kafka-connect StrimziPodSet.

Steps to reproduce

KafkaConnect config to reproduce

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: my-connect-cluster
spec:
  template:
    serviceAccount:
      metadata:
        annotations:
          eks.amazonaws.com/role-arn: arn:aws:iam::${AWS_ACCOUNT_ID}:role/my-role
  version: 3.7.0
  replicas: 1
  bootstrapServers: my-cluster-kafka-bootstrap:9092
  config:
    config.storage.replication.factor: -1
    offset.storage.replication.factor: -1
    status.storage.replication.factor: -1

Expected behavior

Cluster Operator creates both the pod and the annotated service account

Strimzi version

0.41.0

Kubernetes version

1.26

Installation method

No response

Infrastructure

No response

Configuration files and logs

No response

Additional context

No response

[scholzj]

Please share the full log from the operator.

[yordanovsstoyan]

First I get

2024-05-17 08:02:27 INFO  AbstractOperator:399 - Reconciliation #1(watch) KafkaConnect(my-namespace/kafka): Reconciliation is in progress

Then

2024-05-17 07:59:38 ERROR StrimziPodSetController:409 - Reconciliation #7(watch) StrimziPodSet(my-namespace/kafka-connect): StrimziPodSet kafka-connect in namespace my-namespace reconciliation fail │
│ io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.                                                                                                                                          │
│     at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:129) ~[io.fabric8.kubernetes-client-api-6.12.1.jar:?]                                                     │
│     at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:122) ~[io.fabric8.kubernetes-client-api-6.12.1.jar:?]                                                     │
│     at io.fabric8.kubernetes.client.dsl.internal.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:44) ~[io.fabric8.kubernetes-client-6.12.1.jar:?]                                                   │
│     at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.create(BaseOperation.java:1155) ~[io.fabric8.kubernetes-client-6.12.1.jar:?]                                                                             │
│     at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.create(BaseOperation.java:98) ~[io.fabric8.kubernetes-client-6.12.1.jar:?]                                                                               │
│     at io.strimzi.operator.cluster.operator.assembly.StrimziPodSetController.maybeCreateOrPatchPod(StrimziPodSetController.java:470) ~[io.strimzi.cluster-operator-0.41.0.jar:0.41.0]                                   │
│     at io.strimzi.operator.cluster.operator.assembly.StrimziPodSetController.reconcile(StrimziPodSetController.java:398) ~[io.strimzi.cluster-operator-0.41.0.jar:0.41.0]                                               │
│     at io.strimzi.operator.cluster.operator.assembly.StrimziPodSetController.run(StrimziPodSetController.java:566) ~[io.strimzi.cluster-operator-0.41.0.jar:0.41.0]                                                     │
│     at java.lang.Thread.run(Thread.java:840) ~[?:?]                                                                                                                                                                     │
│ Caused by: java.io.IOException: request timed out                                                                                                                                                                       │
│     at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:504) ~[io.fabric8.kubernetes-client-6.12.1.jar:?]                                                                 │
│     at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:524) ~[io.fabric8.kubernetes-client-6.12.1.jar:?]                                                                │
│     at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleCreate(OperationSupport.java:340) ~[io.fabric8.kubernetes-client-6.12.1.jar:?]                                                                  │
│     at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleCreate(BaseOperation.java:754) ~[io.fabric8.kubernetes-client-6.12.1.jar:?]                                                                        │
│     at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleCreate(BaseOperation.java:98) ~[io.fabric8.kubernetes-client-6.12.1.jar:?]                                                                         │
│     at io.fabric8.kubernetes.client.dsl.internal.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:42) ~[io.fabric8.kubernetes-client-6.12.1.jar:?]                                                   │
│     ... 6 more                                                                                                                                                                                                          │
│ Caused by: java.net.http.HttpTimeoutException: request timed out                                                                                                                                                        │
│     at jdk.internal.net.http.ResponseTimerEvent.handle(ResponseTimerEvent.java:63) ~[java.net.http:?]                                                                                                                   │
│     at jdk.internal.net.http.HttpClientImpl.purgeTimeoutsAndReturnNextDeadline(HttpClientImpl.java:1270) ~[java.net.http:?]                                                                                             │
│     at jdk.internal.net.http.HttpClientImpl$SelectorManager.run(HttpClientImpl.java:899) ~[java.net.http:?]

[yordanovsstoyan]

Is this log enough ?

[yordanovsstoyan]

We are trying to authenticate kafka-connect with IRSA. Adding the annotation to serviceAccount template in the KafkaConnect CRD, does not create any pods at all. If the template for the serviceAccount is removed from the KafkaConnect CRD , the pods get created instantly.

IRSA attaches an aws-iam-token volume to the pod and adds these variables AWS_STS_REGIONAL_ENDPOINTS, AWS_DEFAULT_REGION, AWS_REGION , AWS_ROLE_ARN , AWS_WEB_IDENTITY_TOKEN_FILE to the container in the pod and. It also mounts the aws-iam-token volume to /var/run/secrets/eks.amazonaws.com/serviceaccount

Do you have any idea what might be causing the issue ? We are willing to take a look at the code and open up a PR if needed.

[scholzj]

Why don't you start by sharing the full log as you were asked to?

[yordanovsstoyan]

Thanks for the fast responses and apologies for not sharing the full logs yet, but they partly contain project related information that needs to be removed first. In preparation for cleaning the logs I just deployed it in a very similar environment and surprisingly everything there worked as expected, so it might be an environment specific issue. I will write again when I had some time to look at the differences and also share some more information based on the findings then (including full logs if still needed).

[yordanovsstoyan]

The issue was resolved by increasing the kubernetes request time out. Any idea why this fixes the error, just curious ?

extraEnvs:
    - name: KUBERNETES_REQUEST_TIMEOUT
      value: '60000

[scholzj]

I do not know what the error was. So no idea, sorry.