Support cert-manager based deployments

[mcristina422]

What would you like to be added:

Enable cert-manager enabled clusters to use this webhook

This request is to support webhook certificates via a https://github.com/jetstack/cert-manager ca injector. This will benefit the webhook in a few ways, the certificate will no longer need to be generated by the application itself, removing a dependency on the CSR based upstream k8s certificate manager (where issues do arise like the metrics changes in #92). Additionally, it will remove the need for secret and CSR RBAC, as cert-manager will inject the CA to the webhook at deploy time. The certificate will get rotated in place when it expires

Why is this needed:
The "standard" set by kubebuilder/controller-runtime is cert manager based certificates as it simplifies certificate management
https://book.kubebuilder.io/cronjob-tutorial/cert-manager.html#deploying-the-cert-manager

Both cert-manager based webhooks and CSR webhooks can both be supported at the same time with feature flags and updated RBAC. We run this today internally by updating RBAC, creating a cert-manager.io Certificate, setting the cert-manager.io/inject-ca-from annotation on the webhook, disabling the in cluster flag, and manually setting tls-cert and tls-key to be a mounted secret volume. But I think the server needs to reload the certificate, like in https://github.com/kubernetes-sigs/controller-runtime/pull/421/files or else when the cert expires, the server will still use the certificate it loaded at startup, not the live certificate on the filesystem

EDIT: The certwatcher may get exported in kubernetes-sigs/controller-runtime#1292

[avdhoot]

@mcristina422 Can you please share steps how you did it?

[avdhoot]

Thanks @mcristina422 for hints. Below yaml for how we deployed amazon-eks-pod-identity-webhook with cert-manager

kind: Deployment
metadata:
  name: pod-identity-webhook
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: pod-identity-webhook
  template:
    metadata:
      labels:
        app: pod-identity-webhook
    spec:
      serviceAccountName: pod-identity-webhook
      containers:
        - name: pod-identity-webhook
          image: amazon/amazon-eks-pod-identity-webhook:2c44edc
          imagePullPolicy: Always
          command:
            - /webhook
            - --in-cluster=false
            - --namespace=kube-system
            - --service-name=pod-identity-webhook
            - --annotation-prefix=eks.amazonaws.com
            - --token-audience=sts.amazonaws.com
            - --logtostderr
          volumeMounts:
            - name: cert
              mountPath: "/etc/webhook/certs"
              readOnly: true
      volumes:
        - name: cert
          secret:
            secretName: pod-identity-webhook-cert
---
apiVersion: cert-manager.io/v1alpha3
kind: Certificate
metadata:
  name: pod-identity-webhook
  namespace: kube-system
spec:
  secretName: pod-identity-webhook-cert
  commonName: "pod-identity-webhook.kube-system.svc"
  dnsNames:
  - "pod-identity-webhook"
  - "pod-identity-webhook.kube-system"
  - "pod-identity-webhook.kube-system.svc"
  - "pod-identity-webhook.kube-system.svc.local"
  isCA: true
  duration: 43800h
  renewBefore: 8760h
  issuerRef:
    name: selfsigned
    kind: ClusterIssuer
---
apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
  name: pod-identity-webhook
  namespace: kube-system
  annotations:
    cert-manager.io/inject-ca-from: kube-system/pod-identity-webhook
webhooks:
  - name: pod-identity-webhook.amazonaws.com
    failurePolicy: Ignore
    clientConfig:
      service:
        name: pod-identity-webhook
        namespace: kube-system
        path: "/mutate"
    rules:
      - operations: [ "CREATE" ]
        apiGroups: [""]
        apiVersions: ["v1"]
        resources: ["pods"]

[christopherhein]

In case you are interested in reviewing the implementation - kubernetes-sigs/controller-runtime#1441 hopefully this fulfils your needs as well.

[patjones]

Wanted to bump this issue as well thank @mcristina422 and @avdhoot for the cert-manager integration above! I wanted to ask in the meantime how people are getting around the issue of the cert being loaded on start-up? Deleting the webhook pod with a job periodically? I see the above configuration is setting a long duration for the cert.

[mcristina422]

On our clusters we use a shorter duration cert and we automatically terminate our EKS nodes weekly, so pods will get rolled out frequently enough to get new certificates. A job to rollout the deployment makes sense to me!

Long term the plan should be to refactor this webhook to use this package to automatically reload the certificate

[avdhoot]

Long duration for the cert is short term fix for cert rotation issue If the app does not reload when cert change.

…

On Thu, May 13, 2021 at 12:28 AM Michael Cristina ***@***.***> wrote: On our clusters we use a shorter duration cert and we automatically terminate our EKS nodes weekly, so pods will get rolled out frequently enough to get new certificates. A job to rollout the deployment makes sense to me! Long term the plan should be to refactor this webhook to use this <kubernetes-sigs/controller-runtime#1441> package to automatically reload the certificate — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#94 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKA6Q4JBUHGDAUYWW6EASLTNLFVLANCNFSM4T34I6LQ> .

[danopia]

This unrelated PR might help with the renewal story? #87

This change adds support for the --external-tls-renewal flag which will make the k8s Secret "readonly", i.e only read but never update it. Instead of having the k8s go-client certificate_manager.Manager renew it via CSR API, it will just try to re-read it from the secret when it is is nearing expiration time.