Replace usage of static client credentials for components running in seed talking to the shoot API server

[rfranzke]

How to categorize this issue?

/area security
/kind enhancement
/priority 2

What would you like to be added:

Today, all of the components running in the seed clusters and talking to the shoot's API server use a kubeconfig with a static client certificate issued by the cluster's certificate authority (example).

Instead, let's start using similar concepts compared to projected ServiceAccount tokens supported since Kubernetes v1.12.

Concretely, we have to use the TokenRequest API of the shoot's API server to issue short-lived tokens for the respective components.
The tokens should be stored in dedicated Secrets in the shoot namespace in the seed cluster so that they can be mounted to the respective Pods.
Similar to how the kubelet issues a new token when the lifetime reaches 80%, we should refresh such tokens regularly and update them in the respective Secrets in the seed.

This also allows us to use the same kubeconfig for all components which plays well together with what is proposed in #3598.
Also, it helps reducing the load of the Garden cluster since no client certificates/kubeconfigs would be generated anymore for such components, hence, reducing the size of the ShootState and also the size of the etcd backup.

For example, the following kubeconfig could be generated by gardenlet and stored in the kubeconfig Secret in the shoot namespace:

apiVersion: v1
kind: Secret
metadata:
  name: kubeconfig
  namespace: shoot--foo--bar
type: Opaque
stringData:
  kubeconfig.yaml: |
    apiVersion: v1
    kind: Config
    current-context: shoot--foo--bar
    clusters:
    - cluster:
        server: https://kube-apiserver
      name: shoot--foo--bar
    contexts:
    - name: shoot--foo--bar
      context:
        user: shoot--foo--bar
        cluster: shoot--foo--bar
    users:
    - name: shoot--foo--bar
      user:
        tokenFile: /var/run/secrets/gardener.cloud/shoot/token

Also, let's say the short-lived token was stored in the some-component-shoot-token Secret in the shot namespace:

apiVersion: v1
kind: Secret
metadata:
  name: some-component-shoot-token
  namespace: shoot--foo--bar
type: Opaque
stringData:
  token: some-token

Then the PodSpec would look as follows:

apiVersion: v1
kind: Pod
spec:
  volumes:
  - name: kubeconfig
    secret:
      secretName: kubeconfig
  - name: shoot-token
    secret:
      secretName: some-component-shoot-token
  containers:
  - name: foo
    volumeMounts:
    - name: kubeconfig
      mountPath: /var/run/secrets/gardener.cloud/kubeconfig
      readOnly: true
    - name: shoot-token
      mountPath: /var/run/secrets/gardener.cloud/shoot
      readOnly: true
    command:
    - /foo
    - --kubeconfig=/var/run/secrets/gardener.cloud/kubeconfig/kubeconfig.yaml
...

Why is this needed:

In order to eliminate static credentials requiring costly manual rotation which doesn't scale.
Projected ServiceAccount tokens can be configured with expirationSeconds and are regularly auto-rotated and re-mounted by the kubelet without further do.
Clients are expected to regularly reload the token, however, the client-go supports reloading the tokens specified in tokenFiles since v10.0.0, (see kubernetes/kubernetes#70606), and most of the components are either already using this version or could be updated.

Target Picture:

New in this picture is: Secret with short-lived token. Also the pod spec is adapted to mount the token and the component-kubeconfig points to the mounted tokenFile.
In the Shoot the ServiceAccount needs to be added and referenced by the existing rolebinding.

Open Questions:

• Do all components support auto reloading tokens from disk -> Seems like yes supported in client-go since v1.13.0.
• Does the kubelet auto-remount secrets if the data of the secret changes -> yes
• Do we want to reuse a component for our controller (like the grm) or create a new component which includes this token functionality. -> we use the grm for now
• How is the controller itself authenticated against the KAPI?
• Adapt all components running in the Seed targeting the Shoot to use a non-static token.
• We create a significant amount of API requests on the Seed API server when updating tokens of all components in the Shoot namespaces upon token expiration. Will this have scaling performance implications on the Seed APIServer?
• How often do we want to rotate the tokens? What to choose for expiration time?
• A TokenRequest also includes an audiences. We are currently unsure what this field does.

Tasks:

• Prepare gardener-resource-manager

  • Move github.com/gardener/gardener-resource-manager into gardener/gardener #4757
  • Add static ServiceAccount token invalidator #4817
  • Add ServiceAccount token requestor controller #4867
  • Enhance TokenRequestor to be able to sync tokens to a Secret in the target cluster #5084

• Adapt gardener/gardener components

  • Enable TokenRequestor and eliminate client certificate for kube-scheduler #4931
  • Eliminate client certificate for kube-controller-manager and cluster-autoscaler #5007
  • Eliminate client certificate for kube-state-metrics and prometheus #5008
  • Eliminate client certificate for vpa-{admission-controller,recommender,updater} #5009
  • Eliminate client certificate for kube-rbac-proxy #5010
  • Eliminate client certificate for gardenlet and extension controllers #5012
  • 🛡 Eliminate client certificate for cloud-config-downloader #5121
  • 🛡 Eliminate client certificate for gardener-resource-manager #5138
  • 🛡 Eliminate static token for promtail #5153
  • 🛡 Eliminate client certificate for dependency-watchdog-probe #5685
  • 🛡 Preparations for allowing extension controllers eliminating static credentials #5162
  • 🛡 Adapt generic {Worker,ControlPlane} actuators and terraformer library for elimination of static credentials #5163

• Adapt gardener/gardener-extension-* components

  • shoot-cert-service: 🛡 Revendor gardener/gardener@v1.39, update Golang version to 1.17, enable ServiceAccount token projection and token requestor gardener-extension-shoot-cert-service#103
  • shoot-dns-service: 🛡 Revendor gardener/gardener@v1.39, update Golang version to 1.17, enable ServiceAccount token projection and token requestor gardener-extension-shoot-dns-service#96
  • shoot-oidc-service: 🛡 Revendor gardener/gardener@v1.39, update Golang version to 1.17, enable ServiceAccount token projection gardener-extension-shoot-oidc-service#16
  • provider-alicloud: 🛡 Enable ServiceAccount token projection and token requestor gardener-extension-provider-alicloud#405
  • provider-aws: 🛡 Enable ServiceAccount token projection and token requestor gardener-extension-provider-aws#467
  • provider-azure: 🛡 Enable ServiceAccount token projection and token requestor gardener-extension-provider-azure#421
  • provider-gcp: 🛡 Enable ServiceAccount token projection and token requestor gardener-extension-provider-gcp#368
  • provider-openstack: 🛡 Enable ServiceAccount token projection and token requestor gardener-extension-provider-openstack#369
  • provider-vsphere: 🛡 Enable ServiceAccount token projection and token requestor gardener-attic/gardener-extension-provider-vsphere#192
  • provider-equinix-metal: 🛡 Enable ServiceAccount token projection and token requestor gardener-extension-provider-equinix-metal#182
  • external-dns-management: 🛡 Enable ServiceAccount token projection external-dns-management#237

[rfranzke]

/assign @BeckerMax @rfranzke
/in-progress

[jia-jerry]

@rfranzke , I like this pic you drew because it help us understand how to rotate shoot kubeconfig token easily. Do you have plan to add this token rotation process in the gardener doc?

[rfranzke]

@jia-jerry There are already other, probably more detailed pics in the docs: https://github.com/gardener/gardener/blob/master/docs/concepts/resource-manager.md#tokenrequestor