Ingress annotations not propagated to the generated Kubernetes service object

[project-administrator]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

Ingress resource managed by the cilium ingressClass creates the kubernetes service. According to the Cilium Ingress Documentation it should pass these annotation to the Kubernetes service object: service.beta.kubernetes.io, service.kubernetes.io, cloud.google.com

However, after testing this on the EKS v1.28.8 cluster it turns out that only this annotation is being passed to the service object: cloud.google.com.

Other two annotations are not passed (service.beta.kubernetes.io, service.kubernetes.io) despite the fact that the cilium-operator runs with these parameters: --ingress-lb-annotation-prefixes='service.beta.kubernetes.io,service.kubernetes.io,cloud.google.com'.

Cilium Version

cilium-cli: v0.16.7 compiled with go1.22.2 on linux/amd64
cilium image (default): v1.15.4
cilium image (stable): v1.15.4
cilium image (running): 1.15.1

Kernel Version

6.1.84

Kubernetes Version

v1.28.8-eks-adc7111

Regression

No response

Sysdump

No response

Relevant log output

# cilium-operator reconciliation messages after I added the annotation. No mentions about the Kubernetes service object:

time="2024-05-10T12:36:14Z" level=debug msg="Found managed pod due to presence of a CEP" identity=0 k8sPodName=kube-system/coredns-cfb945f66-fj855 subsys=cilium-operator-aws
time="2024-05-10T12:36:14Z" level=debug msg="Found managed pod due to presence of a CEP" identity=0 k8sPodName=kube-system/coredns-cfb945f66-c7cqj subsys=cilium-operator-aws
time="2024-05-10T12:36:14Z" level=debug msg="Found managed pod due to presence of a CEP" identity=0 k8sPodName=kube-system/coredns-cfb945f66-fkkl5 subsys=cilium-operator-aws
time="2024-05-10T12:36:14Z" level=debug msg="Controller func execution time: 81.219µs" name=restart-unmanaged-pods subsys=controller uuid=b194022c-d3a1-4921-a568-d6cc9ad74071
time="2024-05-10T12:36:19Z" level=info msg="Reconciling Ingress" controller=ingress resource=cilium-svc-mesh-test/basic-ingress subsys=ingress
time="2024-05-10T12:36:19Z" level=debug msg="Updating dedicated resources" controller=ingress resource=cilium-svc-mesh-test/basic-ingress subsys=ingress
time="2024-05-10T12:36:19Z" level=debug msg="CiliumEnvoyConfig cilium-svc-mesh-test/cilium-ingress-cilium-svc-mesh-test-basic-ingress has been unchanged" subsys=ingress
time="2024-05-10T12:36:19Z" level=debug msg="Service cilium-svc-mesh-test/cilium-ingress-basic-ingress has been updated" subsys=ingress
time="2024-05-10T12:36:19Z" level=info msg="Starting reconciliation" controller=ciliumenvoyconfig resource=cilium-svc-mesh-test/cilium-ingress-basic-ingress subsys=ciliumenvoyconfig
time="2024-05-10T12:36:19Z" level=info msg="Successfully reconciled" controller=ciliumenvoyconfig resource=cilium-svc-mesh-test/cilium-ingress-basic-ingress subsys=ciliumenvoyconfig
time="2024-05-10T12:36:19Z" level=debug msg="Endpoints cilium-svc-mesh-test/cilium-ingress-basic-ingress has been updated" subsys=ingress
time="2024-05-10T12:36:19Z" level=debug msg="Trying to cleanup potentially existing shared resources" controller=ingress resource=cilium-svc-mesh-test/basic-ingress subsys=ingress
time="2024-05-10T12:36:19Z" level=debug msg="CiliumEnvoyConfig cilium/cilium-ingress has been unchanged" subsys=ingress
time="2024-05-10T12:36:19Z" level=debug msg="Updating Ingress status" controller=ingress resource=cilium-svc-mesh-test/basic-ingress subsys=ingress
time="2024-05-10T12:36:19Z" level=info msg="Successfully reconciled Ingress" controller=ingress resource=cilium-svc-mesh-test/basic-ingress subsys=ingress
time="2024-05-10T12:36:19Z" level=info msg="Reconciling Ingress" controller=ingress resource=cilium-svc-mesh-test/basic-ingress subsys=ingress
time="2024-05-10T12:36:19Z" level=debug msg="Updating dedicated resources" controller=ingress resource=cilium-svc-mesh-test/basic-ingress subsys=ingress
time="2024-05-10T12:36:19Z" level=debug msg="CiliumEnvoyConfig cilium-svc-mesh-test/cilium-ingress-cilium-svc-mesh-test-basic-ingress has been unchanged" subsys=ingress
time="2024-05-10T12:36:19Z" level=debug msg="Service cilium-svc-mesh-test/cilium-ingress-basic-ingress has been updated" subsys=ingress
time="2024-05-10T12:36:19Z" level=debug msg="Endpoints cilium-svc-mesh-test/cilium-ingress-basic-ingress has been updated" subsys=ingress
time="2024-05-10T12:36:19Z" level=debug msg="Trying to cleanup potentially existing shared resources" controller=ingress resource=cilium-svc-mesh-test/basic-ingress subsys=ingress
time="2024-05-10T12:36:19Z" level=debug msg="CiliumEnvoyConfig cilium/cilium-ingress has been unchanged" subsys=ingress
time="2024-05-10T12:36:19Z" level=debug msg="Updating Ingress status" controller=ingress resource=cilium-svc-mesh-test/basic-ingress subsys=ingress
time="2024-05-10T12:36:19Z" level=info msg="Successfully reconciled Ingress" controller=ingress resource=cilium-svc-mesh-test/basic-ingress subsys=ingress
time="2024-05-10T12:36:21Z" level=debug msg="Controller func execution time: 1.277122ms" name=k8s-heartbeat subsys=controller uuid=f6017fc8-faac-4a45-910e-b451fd1bc725
time="2024-05-10T12:36:21Z" level=debug msg="Controller func execution time: 4.140447ms" name=k8s-heartbeat subsys=controller uuid=f869e9a3-cca0-429b-bb6c-96fe11c5ebba
time="2024-05-10T12:36:22Z" level=debug msg="Controller func execution time: 3.243761ms" name=k8s-heartbeat subsys=controller uuid=ae50f546-d025-43ac-b51a-e142959b370f

Anything else?

How to reproduce
Create a test ingress with the annotations, e.g.:

# Basic ingress for istio bookinfo demo application, which can be found in below
# https://raw.githubusercontent.com/istio/istio/release-1.11/samples/bookinfo/platform/kube/bookinfo.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: basic-ingress
  annotations:
    service.beta.kuberentes.io/aws-load-balancer-scheme: internal
    service.kuberentes.io/aws-load-balancer-scheme: internal
    cloud.google.com: test
    no-meaning: value1111
#  namespace: default
spec:
  ingressClassName: cilium
  rules:
  - http:
      paths:
      - backend:
          service:
            name: details
            port:
              number: 9080
        path: /details
        pathType: Prefix
      - backend:
          service:
            name: productpage
            port:
              number: 9080
        path: /
        pathType: Prefix

After it's processed by the cilium-operator, Kubernetes service is created which is supposed to have all three annotations. However, it has only one:

$ kubectl get svc cilium-ingress-basic-ingress -o yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud.google.com: test
  creationTimestamp: "2024-05-10T11:19:34Z"
...

Cilium Users Document

• Are you a user of Cilium? Please add yourself to the Users doc

Code of Conduct

• I agree to follow this project's Code of Conduct

[squeed]

In the example you provided, note that there is a typo, "kuberentes" instead of "kubernetes". Can you double-check your config?