Is there a good solution/tool to generate a new helm chart from the manifests?

[dashanji]

Hi, there, is there a good solution/tool to create a new helm chart from the kustomize manifests generated by kubebuilder?

[dashanji]

Thanks @camilamacedo86. In case anyone else has a similar problem, you can use this helmify tool to convert a bundle yaml to a helm chart.

[vsoch]

We have an automated build pipeline that builds all the recommended artifacts and uploads to GitHub packages: https://github.com/flux-framework/flux-operator/blob/main/.github/workflows/build-deploy.yaml. A caveat - we are still heavily in development and haven't gotten to test a production deployment beyond once or twice! E.g., https://github.com/flux-framework/flux-operator/pkgs/container/flux-operator-bundle. I can give more feedback down the line when we are scaling our test workflows that use the operator to production clusters.

[vbehar]

hi,

on our side we’re using https://github.com/arttor/helmify to generate 2 charts from the kustomize config: 1 for the CRDs, and 1 for the operator. it works pretty well, even if that makes the workflow more complex than it should...

in config/default/kustomization.yaml we removed the reference to the crd dir from the bases field:

bases:
#- ../crd # don't render CRDs by default, because we want to manage them in a specific Helm chart
- ../rbac
- ../manager
- ../webhook
- ../certmanager

and then we’re running the following commands:

$ go run sigs.k8s.io/kustomize/kustomize/v4 build config/default | go run github.com/arttor/helmify/cmd/helmify config/helm-charts/your-operator
$ go run sigs.k8s.io/kustomize/kustomize/v4 build config/crd | go run github.com/arttor/helmify/cmd/helmify config/helm-charts/your-operator-crd

[LCaparelli]

We currently create our helm charts based on an internal template and manually maintain it, with a few hacks here and there to copy stuff over from config/.

[Jay-Madden]

An officially supported helm scaffold option in place of kustomize would be very cool, especially for those of us with needs to deploy to multiple unknown envs

[camilamacedo86]

Hi @dashanji, @Jay-Madden, @LCaparelli, @vbehar,

Firstly, I appreciate the insights you've shared regarding the use of Helm charts for distributing Operators. I'm aiming to understand the core need for this approach.

Looking at successful projects like prometheus-operator, they've opted for a different distribution method. Here's a potential alternative solution:

1. Publish the Operator Image: The same way it's done when building the Helm chart. This image can reside in a registry.
2. Distribute Configuration YAMLs: The main distribution requirement seems to be the YAML files for configuration. (same result of run make install and make deploy IMG=)
3. Single Command Deployment: Create a consolidated YAML file, much like what make install or make deploy IMG=myregistry/myoperator:tag accomplishes. This can allow users to deploy the operator with a straightforward kubectl apply -f <url_to_raw_yaml>.

My question is: wouldn't the above approach meet your distribution needs? If Kubebuilder could generate such a consolidated YAML and host it on GitHub, users could simply apply it as:

kubectl apply -f https://raw.githubusercontent.com/myorg/myproject/master/deploy/install.yaml

So, if kubebuilder project the content for the install.yaml with all config for you just have it your repo would not the need to be addressed?

It seems a majority of the solutions revolve around converting Kustomize manifests to Helm charts. While tools like helmify are handy, I'm wondering if the complexity they introduce is a necessity. Especially if the alternative method mentioned above can achieve the same outcome.

I'm keen to understand any specific advantages Helm offers in this context, or if there are limitations or challenges with the proposed approach.

Your feedback will be invaluable!
Thanks!

[LCaparelli]

I would add though, that the one-off manifest does help open-source controllers to be easily tested out and I think it's a great addition to kubebuilder. It just gets a bit grittier when values should vary between different deployments, which in our case is most frequent.

[camilamacedo86]

HI @LCaparelli, @Jay-Madden

Different clusters need different behaviors (eg, each cluster lives in a different account, with own DNS hosted zone. If our controllers need to know which Hosted Zone it manages, we will need an env var set up differently in each cluster).

Can not it be addressed by env vars or CRD specs?
You also need to use Helm values to provide this info. Am right?
How do you do in the Helm with ArgoCD?

Helm makes that easier for us. Much like Kustomize would as well, I believe, but we have 0 adoption of kustomize-based configuration management. We don't currently believe setting up all infra for Kustomize-based stuff just for our controllers/operators would be worth it, given that everything else we use is Helm-based, including other controllers we don't maintain and all of our apps.

So, the point is how to integrate the solution with ArgoCD, Am right?
Because other solutions, for example, Terraform would work with kustomize.
Please let me know if I missed something

[camilamacedo86]

Hi @Jay-Madden,

Due to the fact that we have "external internal" teams consuming the operator we need a way to parameterize different annotations and labels at install time as the operator is installed against many clusters and needs to be tagged correctly for our internal metrics/telemetry and alerting.

Would not the option to add the labels to all resources sorted out this need, see:

kubebuilder/testdata/project-v4/config/default/kustomization.yaml

Lines 11 to 15 in f474467

	# Labels to add to all resources and selectors.
	#labels:
	#- includeSelectors: true
	# pairs:
	# someName: someValue

I think that in terms of "whats standard" in the Kubernetes community, operator installs are by and large facilitated by Helm charts. So large platform teams are already tooled to deal with charts, perhaps not so much for other distribution methods.

That is really an interesting point of view.
We might be able to create a proposal for an alpha command in kubebuilder that does that.
So that we start to discuss this topic

[LCaparelli]

@camilamacedo86

Can not it be addressed by env vars or CRD specs?

CRDs, I'm not sure it would be a good idea to have different CRDs per environment (not sure that's what you meant). Env vars yes, that's how we do it. That's how we control different configs in different environments.

You also need to use Helm values to provide this info. Am right?

We could just have plain old manifests. The main point here, I think, is not a matter of can, more of "wouldn't it be nice if...". Helm makes this easier. Makes sure we don't duplicate all kinds of things everywhere while still allowing for a reduced set of inputs: we only expose in values.yaml files the configs that variate between environments (replicas, image tag, etc, etc). It's simply more maintainable than naked manifests.

EDIT: which is not to say kustomize isn't as maintainable as Helm, but we surely don't perceive Kustomize to be as widely used for this as Helm. Which might not be strong enough of a reason for Kubebuilder to adopt it, but is good enough for us to manually/hackishly port the kustomize generated stuff into Helm charts ourselves.

So, the point is how to integrate the solution with ArgoCD, Am right?
Because other solutions, for example, Terraform would work with kustomize.
Please let me know if I missed something

Well, no. I think ArgoCD supports kustomize out of the box. Thing is, our entire organization adopted Helm for everything that needs to be deployed on K8s. I think @Jay-Madden put it quite nicely in his comment:

I think that in terms of "whats standard" in the Kubernetes community, operator installs are by and large facilitated by Helm charts. So large platform teams are already tooled to deal with charts, perhaps not so much for other distribution methods.

[Jay-Madden]

Can not it be addressed by env vars or CRD specs?
You also need to use Helm values to provide this info. Am right?
How do you do in the Helm with ArgoCD?

I'm not fully following this train of thought. The annotations and labels need to be present on the controller-manager manifests. Not on the manifests that the operator owns.

[dashanji]

Hi, @camilamacedo86, thanks for the detail about the newest progress. Actually, what we want just is a way/tool that can generate diverse distributions such as kustomize manifests, helm charts, and so on.

As for the new feature, if I understand correctly, it bundles the kustomize directory into one YAML so that users can apply it from an URL. It looks like the kubectl apply -k https://github.com/myorg/myproject/k8s/config can achieve the same effect.

The advantage of Helm over Kustomize is that it can control the specific fields and which resources are created, i,e, sometimes I need to specify the image or imagePullPolicy or serviceAccount and so on, and sometimes I can just install controllers or webhooks or CRDs.

Also, the operator depends on the cert-manager. As a result, we need to install the cert-manager first, and then deploy the operator after that. Using the helm chart can build dependencies into a helm package and it's easy for users to deploy.

[camilamacedo86]

Hi @dashanji

The advantage of Helm over Kustomize is that it can control the specific fields and which resources are created, i,e, sometimes I need to specify the image or imagePullPolicy or serviceAccount and so on, and sometimes I can just install controllers or webhooks or CRDs.

It seems also achievable without Helm at all.

** My intention isn't to dismiss the idea. Rather, I'm keen on understanding the requirements and the reasons behind the preference..**

Achieving Customizability with Operators:

1. Image, ImagePullPolicy, ServiceAccount, etc.:

• CRD Spec: Your Custom Resource (CR) spec can include fields like image, imagePullPolicy, etc. The Operator observes changes to this CR and updates the associated Kubernetes resources.

  apiVersion: example.com/v1
  kind: MyApp
  spec:
    image: "nginx:1.19"
    imagePullPolicy: Always

Environment Variables:

For those who'd prefer not to expose such details in the CRD spec, the Operator can also be configured via environment variables during its deployment.

2. Selective Installation of Controllers, Webhooks, CRDs:

Operators can contain logic to conditionally create certain resources based on the CR's spec. For instance, if a webhooks field is set to true, the Operator ensures the webhooks are deployed.

While Helm excels in packaging and deploying applications, Operators shine when there's a requirement for deeper, application-specific operational logic. For customizability in Kubernetes, Operators offer a more flexible (though complex) approach to capture and automate nuanced operational knowledge.

Also, the operator depends on the cert-manager. As a result, we need to install the cert-manager first, and then deploy the operator after that. Using the helm chart can build dependencies into a helm package and it's easy for users to deploy.

it is a very good point and we could add it to the install.yaml proposed a solution as well when the cert-manager is enabled: #3626

Let's think about that.

[dashanji]

Achieving Customizability with Operators:

Do you mean creating operator manifests as templates, then using a top-level CRD to render these templates for customizing fields and selecting resources for installation?

[dashanji]

As for the cert-manager, it looks like the official only supports the helm installation via the repo directory. We may need some integrations with helm cli.

[camilamacedo86]

Hi @dashanji, @Jay-Madden, @LCaparelli, @vbehar,

I'm contemplating the possibility of creating a proposal for an alpha command in kubebuilder to address the need to generate a helm chart from the scaffold builds with Kuebuilder. I believe it's worth opening up a discussion on this.

However, before moving forward, it's imperative that we articulate all the reasons for introducing such a command. It's essential to delve deep into the genuine scenarios and needs, as there might be alternative solutions available.

While I can't guarantee its acceptance into kubebuilder or promise my endorsement at this stage, I believe that the initial step should be to craft a well-thought-out proposal. This document should lucidly detail our motivations and rationale. This will provide a solid base for discussion and enable input from other members of the community. From there, we can further refine our approach, consider additional alternatives, and possibly discover further improvements.

For reference on how to structure our proposal, here's the template: Kubebuilder Design Proposal Template. PLease, feel free to try to push a PR for we further discuss it.

Feel free to adjust it further based on your specific needs or context!

[camilamacedo86]

Hi @dashanji, @Jay-Madden, @LCaparelli, @vbehar,

Also, it's worth noting that ArgoCD has built-in support for Kustomize. Given this, I'm curious as to why you're opting to generate Helm charts instead of using Kustomize directly. Is it possible that the requirement stems from not fully leveraging ArgoCD's capabilities? Have you considered configuring ArgoCD to deploy a project created with Kubebuilder using Kustomize?

See: https://redhat-scholars.github.io/argocd-tutorial/argocd-tutorial/03-kustomize.html

[LCaparelli]

That sounds great, thanks for moving this forward and incentivizing discussion.

I'd say it's not a matter of ArgoCD supporting Kustomize or not, but acknowledging the fact many organizations prefer Helm over Kustomize, and whether this is something the project would like to facilitate for its users. Wdyt?

[Jay-Madden]

Yes argo supports kustomize, however the app of apps pattern does not work well for the double level of indirection required to make it work for other repos (argo wont trigger on changes), as well as it does not support versioned deployments. Nor allow for control over the overridden annotations on the manifests.

[camilamacedo86]

Hello @dashanji, @Jay-Madden, @LCaparelli, @vbehar,

Following our discussions and the valuable inputs received, I've identified some key motivations for integrating the feature:

1. Helm Distribution: Allow users who want to distribute their solution to utilize Helm as a package manager.

2. Helm Dependency Management: Cater to those who wish to capitalize on Helm's dependency management capabilities.

3. Precedence in the Ecosystem: It's worth noting that the Prometheus Operator offers a Helm Chart for distributing their Operator Solution. (We'll need to verify and validate this information). In our proposal, it would be beneficial to list other prominent projects that have adopted a similar approach, allowing their solutions to be installed via Helm.

4. Compatibility with Solutions like FluxCD: Helm-based tools, such as FluxCD, can manage the Operators, providing an integrated solution.

Considering the aforementioned, here are the proposed next steps:

1. Proposal Submission: Create a PR with the proposal. A reference model for this proposal could closely resemble: https://github.com/kubernetes-sigs/kubebuilder/blob/master/designs/helper_to_upgrade_projects_by_rescaffolding.md

2. Feature Status: Introduce this feature as an alpha command. This will allow us to gauge its usability, value, and long-term maintenance overhead.

3. Testing: It's imperative that this feature undergoes extensive e2e testing, akin to how the alpha generate command is tested.

4. Documentation: Ensure comprehensive coverage in our documentation to assist users in understanding and leveraging this feature.

5. Options to add the feature

Command-Based Approach

The command should accept as input:

• The path directory of the Kubebuilder project.
• The path where the resulting Helm Chart will be written based on the provided input.

Thus, by executing a command like:

kubebuilder alpha generate-helm-chart --from=<path> --output=<path>

If --from is not specified, the current directory should be used by default. Similarly, if --output isn't given, the Helm Chart should be generated in the project's directory (e.g., deploy/helm-chart).

Plugin-Based Approach

This implies that commands such as:

kubebuilder init --plugins=helmchart/v1alpha

or

kubebuilder edit --plugins=helmchart/v1alpha

can be executed, resulting in the necessary scaffolding for Helm.

To decide on the optimal approach, it's essential to understand the Helm chart's structure and layout. Based on that, we can determine if a command or plugin approach is more suitable, weighing the pros and cons of each. My current inclination is towards plugins, given the existing structure, but we should evaluate and decide on the most efficient solution.

Therefore, IMO the proposal must have in the details for implementation the description/definition of the Helm-Chart layout. How should be the layout of the HelmChart, if any of you start that and be able to add the motivations and etc. how should be the result (HelmChart) I can contribute with this part regards how to get it implemented in the Kubebuilder.

Thank you for your consideration, and I look forward to your feedback!
Please, your help to achieve this goal is very welcome
And you all have my support on this one.

[dashanji]

Hi @camilamacedo86. Thank you once again for providing these details. Based on my previous experience with Helmify for generating helm charts and handcrafting helm charts through Kustomize manifests, I would like to try to submit the proposal about the layouts of generated helm chart.

[LCaparelli]

I'm happy to help as well, with the proposal and (hopefully) eventual implementation.

[camilamacedo86]

We can work together on this one.
If you are able to just start the doc and how the HelmChart of the Kubebuilder scaffold should be that would be very helpful,

Or if you prefer to add this info here we can either try to get the pieces to work on the proposal.

[camilamacedo86]

Also, good examples to justify the motivation:

• https://artifacthub.io/packages/helm/mongodb-helm-charts/community-operator
• https://cert-manager.io/v1.6-docs/installation/helm/#1-add-the-jetstack-helm-repository
• https://bitnami.com/stack/prometheus-operator/helm
• https://rook.io/docs/rook/v1.12/Helm-Charts/operator-chart/#configuration

[dashanji]

Hi, @camilamacedo86, @Jay-Madden, @LCaparelli, @vbehar. I just submitted a PR #3632 that describes the layout information of the helm chart in my mind. Welcome any comments. Thanks.

[exu]

Great to see the progress in Helm charts support! We're waiting for it in testkube :D

[camilamacedo86]

We want to create a plugin that will generate the scaffold to package the operator using helm
How to do that has been discussed in the PR: #3632

So, I think we can close this one
However, if anyone thinks it should be re-opened, please feel free.