Mutating Webhook is being ignored

[EinavDanielDX]

I have deployed k8s-image-swapper into my cluster and it looks like the webhook is being ignored. No log in webhook server regarding pod creation could be found.

Some details:
Platform: EKS, v1.24
swapper helm chart version: 1.8.0
swapper image version: 1.5.7

Values:

image:
  tag: "1.5.7"
awsSecretName: k8s-image-swapper-aws
config:
  ImageSwapPolicy: "always"
  ImageCopyPolicy: "immediate"
  source:
    filters:
      - jmespath: "contains(container.image, '.dkr.ecr.') && contains(container.image, '.amazonaws.com')"
      - jmespath: "obj.metadata.namespace == 'kube-system'"
  target:
    type: aws
    aws:
      accountId: "my-account-id-number"
      region: eu-west-1
      ecrOptions:
        tags:
          - key: cluster
            value: my-cluster

testing:

# executed:
kubectl apply -f https://k8s.io/examples/pods/simple-pod.yaml

# result
pod/nginx created

logs on webhook server:

{"level":"info","file":"/.k8s-image-swapper.yaml","time":"2023-11-20T18:53:24Z","message":"using config file"}
6:53PM DBG auth token set, schedule next token renewal expiryAt=2023-11-21T06:53:24Z renewalAt=2023-11-21T06:51:24Z
6:53PM INF Listening on :8443

[estahn]

@EinavDanielDX The config appears to be incorrect, e.g. there is no config prefix for ImageSwapPolicy.

Here is an example: https://github.com/estahn/k8s-image-swapper/blob/main/.k8s-image-swapper.yml

I think I need to find a better way of surfacing config issues.

[EinavDanielDX]

Thank you for your response.
My configuration was set according to this helm chart, where I figured config prefix is meant to represent swapper configuration.

Anyway, I am using Kyverno instead now, its a bit more complicated but got the job done.

A note worth mentioning:
When using external CNI in EKS (for example, we are using Calico instead of default EKS CNI), there will be an issue with the communication between the K8S API server and the webhook. in such case it's best to expose the relevant services using

  hostNetwork: true
  dnsPolicy: "ClusterFirstWithHostNet"

Could be helpful to add this note to an installation\configuration guide

[estahn]

Right, I didn't realise it is the helm config 🤦🏼

That is good feedback. I will try to add this to the documentation. I haven't worked with EKS yet, so not sure where exactly this should go.

[EinavDanielDX]

You can read about the issue with EKS here, it is relevant to any webhook service.

So in case a user of k8s-image-swapper is using EKS with 3rd party CNI, he should have the following applied in the deployment corresponding to the webhook

hostNetwork: "true"
dnsPolicy: ClusterFirstWithHostNet

Kyverno had documented it pretty well in their helm chart and I find it most convenient:

  # -- Change `hostNetwork` to `true` when you want the pod to share its host's network namespace.
  # Useful for situations like when you end up dealing with a custom CNI over Amazon EKS.
  # Update the `dnsPolicy` accordingly as well to suit the host network mode.
  hostNetwork: false

  # -- `dnsPolicy` determines the manner in which DNS resolution happens in the cluster.
  # In case of `hostNetwork: true`, usually, the `dnsPolicy` is suitable to be `ClusterFirstWithHostNet`.
  # For further reference: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-dns-policy.
  dnsPolicy: ClusterFirst

[estahn]

@EinavDanielDX Awesome, thanks for that!

I looked at Kyverno a couple of years ago. Out of curiosity, is this a 1:1 replacement for k8s-image-swapper? Any particular issues you encountered?

[EinavDanielDX]

tl'dr:
kyverno + aws pull through cache could replaces the k8s-swapper, but its harder to configure and maintain.

Longer version:
I won't say its a 1:1 replacement, though you can achieve the same results combining Kyverno with AWS pull through cache
I'd say that Kyverno is much more robust solution for mutating whatever Kubernetes objects, not just pods.
It's more complicated to configure and could be used for lots of other scenarios beside swapping images.
Combining this with AWS pull through cache will solve the swapping-image issue for most of the repositories, as we are limited by AWS to use only the following registries:
Docker Hub (registry-1.docker.io)
Kubernetes (registry.8s.io)
Quay (quay.io)
GitHub (ghcr.io)
Azure (*.azurecr.io)

I would have been happy to implement k8s-image-swapper as it is a less complicated solution, though I failed and had to come with an alternative ASAP.

One more feedback I would suggest: With Kyverno I can mutate imagePullSecrets, so I can use my private registry with swapped images, given the secret exists in the namespace. If the pull secret is missing I will be prompted with a warning but the deployment will continue as planned. I think this feature is rather easy to implement as "optional" and will benefit many of this projects' users.