add kubermatic-installer-operator proposal #13327
Conversation
kubermatic-bot
added
docs/none
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
kubermatic-bot
added
the
size/L
| ## Proposal | ||
|
|
||
| A new Kubermatic Installer Operator that utilizes a CRD called `KubermaticInstallation` | ||
| (Since `KubermaticConfiguration` exist).The operator will handle the deployment and management | ||
| of the Kubermatic stack by watching for changes to instances of `KubermaticInstallation` CRD. | ||
|
|
||
| * **KubermaticInstalltion CRD:** Define all necessary configurations needed for the Kubermatic installation | ||
| in a single Kubernetes resource | ||
| * **Kubermatic Installer Operator:** A controller that reacts to changes to KubermaticInstallation resources | ||
| by deploying or updating the Kubermatic stack according to the specified configuration. | ||
| * **Dependencies Management:** The operator will manage dependencies like cert-manager, nginx-ingress-controller, | ||
| and Dex, ensuring they are installed or upgraded as necessary before deploying Kubermatic components. |
There was a problem hiding this comment.
The motivation is good and we've discussed this in SIG Cluster Management numerous time (with some consensus), but in my opinion the proposal needs refinement. Why are we adding another CRD that describes the same thing as a KubermaticConfiguration - a KKP setup?
KKP already has the kubermatic-operator, which is responsible for upgrading all KKP components. What is the benefit of adding an outer operator layer that just exists to run a bit of migration and verification code and then increment the inner operator version, instead of e.g. moving the migration code into the existing operator?
If we get rid of the installer and deliver a Helm chart to install (any) operator, why not create a meta/umbrella chart that installs kubermatic-operator (the existing one) and the dependencies (cert-manager, ingress controller, etc)?
There was a problem hiding this comment.
meta/umbrella chart
Even here I wonder what the benefit is. I think mainly that you keep a consistent set of versions and don't install each latest version of each dependency individually.
Because if the umbrella Chart were to include everything, everyone would complain that it installs too much. So each component would need to be optional anyway, at which point the benefit of the umbrella chart might diminish. Again, besides the good point of having one set of versions blessed by us when we tag a KKP release.
There was a problem hiding this comment.
Thank you for the feedback, I understand the concerns about introducing a new layer when we already have kubermatic-opertor. The key motivation behind proposing an installer operator is to provide a solution that can manage not just the core KKP components, but also additional stacks such as monitoring, logging, backups, etc., which the current kubermatic-operator does not handle. The goal is to offer a unified, automated experience for admins and KKP users
The CRD will not describe the same thing as KubermaticConfiguration, but the other stacks
There was a problem hiding this comment.
I don't think this is the way to go, but just for the sake of poking this argument:
Why include all the things already configurable by KubermaticConfiguration if what you want to focus on are actually just the dependencies? What is the benefit of having all of those KKP settings effectively doubled? Wouldn't it make more sense to have a DefaultStack resource or something that we add to the existing kubermatic-operator and give it the capability to deploy dependencies alongside reconciling KubermaticConfiguration?
In addition, I think the question regarding the benefit of this over providing a meta Helm chart also still stands.
|
|
||
| An administrator wishes to install or upgrade their Kubermatic system:: | ||
|
|
||
| 1. The admin prepares a `KubermaticInstallation` manifest. The goal here is to avoid splitting the configuration and customization to be split into different places and manifests like Helm `values.yaml` file, `KubermaticConfiguration` etc |
There was a problem hiding this comment.
How does the KubermaticInstallation CRD (and the operator) get into the cluster in the first place?
There was a problem hiding this comment.
I was thinking of making the operator very simple so it could be installed with one yaml manifest that contains the CRD, deployment and necessary RBAC permissions. This approach would enable a one-command installation, simplifying the initial setup
There was a problem hiding this comment.
Okay, but my point here was that the installation process is missing a step (installing this new operator) which makes it look more simple than it actually is. We should strive for completeness when describing admin UX and workflows.
| seed: | ||
| - name: seed-1 | ||
| etcdBackupRestore: | ||
| defaultDestination: minio-ext-endpoint | ||
| ####################### | ||
| ### SKIP SEED CONFIG ### | ||
| ####################### |
There was a problem hiding this comment.
What do we gain from this "monolith" CRD over storing multiple YAML documents in a kubermatic.yaml which holds the KubermaticConfiguration "master config" object and multiple Seed objects which you then apply?
I mean:
apiVersion: kubermatic.k8c.io/v1
kind: KubermaticConfiguration
metadata:
name: kubermatic
namespace: kubermatic
spec:
[...]
---
apiVersion: kubermatic.k8c.io/v1
kind: Seed
metadata:
name: seed-1
spec:
[...]
---
apiVersion: kubermatic.k8c.io/v1
kind: Seed
metadata:
name: seed-2
spec:
[...]There was a problem hiding this comment.
(assuming we change the way we ship the Seed CRD). But I'd be very sceptical to add fields to one CRD to generate another CRD on the same layer of the KKP installation, just because currently the CRD isn't there at that time.
There was a problem hiding this comment.
The design of the KubermaticInstallation CRD is flexible, allowing you to either define all components within one comprehensive manifest or split them into individual manifests for each stack, depending on your management preferences. It's not mandatory to create a monolithic YAML manifest. you can choose to apply configurations for each stack separately with their own specific specs. This ensures that the CRD can support a variety of deployment strategies
There was a problem hiding this comment.
I don't understand this point. Are you saying you could apply the same KubermaticInstallation multiple times with only specific settings set to configure the components? That doesn't sound very declarative.
|
|
||
| ## Proposal | ||
|
|
||
| A new Kubermatic Installer Operator that utilizes a CRD called `KubermaticInstallation` |
There was a problem hiding this comment.
This sounds like we're re-inventing existing Helm operators that simply watch some CRD and install Helm charts into the cluster. What would be the benefit and effort of writing a custom one?
KKP already has had multiple bugs in its App feature when it tries to automate managing Helm installations. I'm fearful of writing a Helm reconciler myself.
|
/assign |
What this PR does / why we need it:
This is a proposal for kubermatic installer operator
Which issue(s) this PR fixes:
Fixes #
What type of PR is this?
Special notes for your reviewer:
Does this PR introduce a user-facing change? Then add your Release Note here:
Documentation: