Allow mangling the TCP Deployment

[prometherion]

It would be useful to allow the cluster admin to setup for each Tenant Control Plane Deployment these additional components:

• volumes
• volumeMounts
• containers
• initContainers

A proposal of the TCP manifest could be the following:

apiVersion: kamaji.clastix.io/v1alpha1
kind: TenantControlPlane
metadata:
  name: test
  namespace: default
spec:
  controlPlane:
    deployment:
      additionalInitContainers:
      - {} # corev1.Container type
      additionalContainers:
      - {} # corev1.Container type
      additionalVolumes:
      - {} # corev1.Volume type
      additionalVolumeMounts:
        apiServer:
        - {} # corev1.VolumeMount
        scheduler:
        - {} # corev1.VolumeMount
        controllerManager:
        - {} # corev1.VolumeMount

The use case could be an agent running side by side to the Tenant Control Plane pods, mounting secrets required for further integrations, allowing to specify override values for the kube-apiserver configuration files, etc.

[ksankeerth]

Hi @prometherion,

If this is a low-priority task for the core team, Can I give it a try?

Thanks!

[prometherion]

I was planning to get this shipped for v0.2.0 but since we're importing v1.26.0 Kubernetes code-base, there's an error in the API specification that would impact Kamaji when using the core API types for those additional resources.

More context: kubernetes-sigs/controller-tools#753

We should wait for an upstream fix before getting this done.

[maruina]

Hey @ksankeerth, did you pick this up? I was considering myself giving a try but don't want to duplicate the effort :)

[ksankeerth]

Hey @ksankeerth, did you pick this up? I was considering myself giving a try but don't want to duplicate the effort :)

Go ahead @maruina. I didn't start any effort for this.

[prometherion]

@maruina did controller-tools solved the issue with x-kubernetes-map-type thing? Otherwise, we're pretty blocked for this since I'd like to reuse the same core API resources.

[maruina]

@prometherion I think https://github.com/kubernetes-sigs/controller-tools/releases/tag/v0.11.1 is the release where they reverted the PR causing issues

[prometherion]

Thanks @maruina, it's yours.

Let's sync privately if you need help on this.

[maxgio92]

Hi @maruina did you have a chance to investigate on this?

I think this feature brings a considerable value. I'm thinking about Kubernetes audit - now that dynamic audit configuration is no longer supported ;( - between different use cases.

[maruina]

Hey @maxgio92, I haven't started either as life got in the way :)
Feel free to take this issue if you have capacity

[prometherion]

@bsctl with the support for additional volume mounts, I think we could partially address the kine support with SQLite, although the PVC creation is still delegated to the cluster administrator.

Don't you think so?

[bsctl]

Also consider SQLite with ephemeral volumes to cover some corner cases as for example ephemeral clusters.