Upgrade Flux to the Kustomize v1beta2 API

[stefanprodan]

Starting with v0.18, Flux comes with a new reconciler based on Kubernetes server-side apply and graduates the Flux Kustomize API to v1beta2.

Breaking changes

• Namespaced objects must contain metadata.namespace, defaulting to the default namespace is no longer supported. Setting a namespace for all objects reconciled by a Kustomization can be done with spec.targetNamespace.

• The logs, events and alerts that report Kubernetes namespaced object changes are now using the Kind/Namespace/Name format instead of Kind/Name.

• The minimum required version of Kubernetes has changed to:

  Kubernetes version	Minimum required
  v1.16	>= 1.16.11
  v1.17	>= 1.17.7
  v1.18	>= 1.18.4
  v1.19 and later	>= 1.19.0

Features and improvements

• Being able to validate and reconcile sources that contain both CRDs and CRs.
• Being able to wait for all the applied resources to become ready without requiring users to fill-in the health check list.
• Improve performance (CPU, memory, network, FD usage) and reduce the number of calls to Kubernetes API by replacing kubectl execs with a specialised applier written in Go.
• Detect and report drift between the desired state (git, s3, etc) and cluster state reliably.
• Improve the overall observably of the reconciliation process by reporting in real-time the garbage collection and health assessment actions.
• Reconcile empty sources including pruning of all the resources previously applied.
• Mask secrets data in logs, events and alerts.

API changes

The kustomize.toolkit.fluxcd.io/v1beta2 API is backwards compatible with v1beta1.

Additions, deprecations and removals:

• .spec.patchesStrategicMerge deprecated in favour of .spec.patches
• .spec.patchesJson6902 deprecated in favour of .spec.patches
• .spec.validation deprecated and no longer used (server-side validation is implicit)
• .spec.wait added (when enabled, will wait for all the reconciled resources to become ready)
• .status.snapshot replaced by .status.inventory

Upgrade procedure

To upgrade your cluster, download the Flux CLI binary from the release page and rerun the flux bootstrap command. This will upgrade the Flux components on your cluster to the latest version. Other upgrade options can be found on the docs website.

To update the Flux Kustomization manifests in Git see the following table:

Change in the new version	What you should do
Version is now v1beta2	Change the version: apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
.spec.validation removed	Server-side validation is now assumed. Remove this field from .spec.
.spec.patchesStrategicMerge deprecated in favour of .spec.patches	Convert each entry from .spec.patchesStrategicMerge into an inline strategic merge patch, like this example given in the Kustomize documentation, and append to .spec.patches.. Note that the value in the patch field is quoted; that is, it is the YAML or JSON of the patch, stringified.
.spec.patchesJson6902 deprecated in favour of .spec.patches	Convert each entry from .spec.patchesJson6902 into an inline JSON6902 patch, and append to .spec.patches. Note that the value in the patch field is quoted; that is, it is the YAML or JSON of the patch, stringified.
.status.snapshot replaced by .status.inventory	.status is not kept in files, so you will not need to account for this.

ℹ️ Note that updating the manifests in Git to v1beta2 can be done at any time after the Flux components upgrade. All users are encouraged to update the manifests as the deprecated fields will be removed when the next API version will be released.

Terrafrom upgrade

When using the Terraform provider for Flux, you have to manually remove the v1beta1 Kustomization from the TF state:

terraform state rm 'kubectl_manifest.sync["kustomize.toolkit.fluxcd.io/v1beta1/kustomization/flux-system/flux-system"]'

[stefanprodan]

⚠️ For those of you using metrics-server, there was a bug in Kubernetes that got fixed here. The fix was back ported in kubernetes/kubernetes#99046 so if you deploy metrics-server (or any other APIService) with a Flux Kustomization, the minimum required version of Kubernetes is:

Kubernetes version	Minimum required
v1.18	>= 1.18.18
v1.19	>= 1.19.10
v1.20	>= 1.20.6
v1.21 and later	>= 1.21.0

Workaround

If you can't upgrade your clusters, then use a Flux HelmRelease to install metrics-server as Helm doesn't uses server-side apply.

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
  name: metrics-server
  namespace: flux-system
spec:
  interval: 60m
  url: https://kubernetes-sigs.github.io/metrics-server/
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: metrics-server
  namespace: flux-system
spec:
  interval: 60m
  targetNamespace: kube-system
  releaseName: metrics-server
  chart:
    spec:
      chart: metrics-server
      sourceRef:
        kind: HelmRepository
        name: metrics-server
  values:
    args:
     - --metric-resolution=15s
     - --kubelet-insecure-tls

[matt-simons]

We've run into an issue with the new v0.18.0 validation related to when a CRD and CR are included in the same Kustomization. However in our scenario the CRD is applied indirectly via an Operator Lifecycle Manager Subscription i.e. installed from https://operatorhub.io/.

In this scenario we can no longer include a Subscription and a CR in the same Kustomization due to the validation failing in the dry-run. In v0.17.0 this was previously possible, it would fail to reconcile all resources on the first apply, then on the second reconcile the CRD would be present and all resources would successfully apply.

The present workaround in v0.18.0 is to split our kustomization into two but it would be great if we could retain the previous functionality. Perhaps a method to override the dry-run or apply all other resources that do not fail their dry-run.

Alternatively maybe it is possible to implement similar validation logic for the Subscription + CR scenario as exists for the CRD + CR scenario.

[texasbobs]

The CRDs+CRs feature works only when the Custom Resource Definition YAMLs are present in the source. If some other controller installs the CRDs then it's not possible for Flux to know that.

This used to work however because you could have validation: none for that particular kustomization. Now that it is validation: server, it fails. We are experiencing this problem now with the k3s system-upgrade-controller. The controller creates the CRD plan at startup. Before this upgrade, it was able to be deployed with Flux2.

[texasbobs]

Any thoughts on adding validation back in, @stefanprodan ?

[stefanprodan]

That's not possible, Kubernetes server-side apply implies validation.

[texasbobs]

That's not possible, Kubernetes server-side apply implies validation.

I'm not following you. The previous versions of flux2 would work with validation: none. Now that .spec.validation is removed, we cannot deploy.

[texasbobs]

With the validation: none, the CRs will not get deployed on the first pass due to the error, but the object that creates the CRD (any other objects) will. Then on the next reconciliation, the CRs will successfully get created.

[rustrial]

What is the rational behind the introduction of .spec.patches as a replacement for .spec.patchesStrategicMerge and .spec.patchesJson6902 (which are now deprecated)? What are the benefits?

The motivation as described in fluxcd/kustomize-controller#364 is at best confusing to me and I do not really understand it. That said, I am not a big fan of the new approach .spec.patches, because it uses inline strings (which must still contain valid YAML) instead of properly nested YAML structures. Such inline strings are very cumbersome as there is no editor or linter support, which makes it hard to proof that they contain valid YAML. This makes it much harder to find syntax errors in YAML resources early (before deploying them).

I know, that I am a bit late to the party. But I don't like the fact that breaking changes and deprecations are introduced lightheartedly into FluxCD's APIs (even if it's still beta). Especially, if the disadvantages of changes out-weight their advantages. Contributing to open source is sometimes a bit frustrating, because once you got your changes merged there is always a risk that they are fully or partly reverted by another contributor (acting in good faith) later, without them realising that they actually remove a feature or at least make it worse.

Maybe we need something like a RFC (change management) process for breaking changes and deprecations, to make sure we care a bit more about this aspects and to ensure:

• that we keep the negative effects for users small
• that advantages and disadvantages of changes are explicitly considered for conscious decision making
• that people do not waste their time implementing features which (a) are then not merged or (b) are reverted accidentally by a later contribution
• ...

Many other open source projects have successfully adopted change management process, e.g. Kubernetes with its KEPs or Rust with its RFC.

What do you think?

[stefanprodan]

@rustrial the deprecation was included in this RFC: fluxcd/kustomize-controller#426

The reason for this is to keep our API footprint low, instead of offering 3 ways to achieve the same thing we settled to one way. No matter what the Flux API offers in terms of kustomize overrides, you can create your own kustomize overlays by placing a kustomization.yaml in your source. Kustomize itself supports a variety of patches, plugins, etc, none of those are affected by our API changes.

[stefanprodan]

I really like to understand why we have to fiddle with inline strings in future instead of nested YAML.

I think of Flux patches as a lost resort when the upstream kustomize overlay can't be adjusted right away. What's your reason for defining patches in Flux Kustomizations?

PS. As we are at beta with the Flux API, deprecations could be rolled back if there is a strong motivation on why we should have 3 ways of specifying in-line patches.

[rustrial]

@stefanprodan All good, no need to rollback. For me it's more about making sure, that in future we keep the amount of breaking changes and deprecation as small as possible and maybe introducing a more formal change management process (e.g. RFCs). Having more and more assets in GitOps, such refactoring are expensive and risky and might cause users to stick around with old FluxCD version, just because they are afraid of refactoring their stuff.

[stefanprodan]

in future we keep the amount of breaking changes and deprecation as small as possible and maybe introducing a more formal change management process (e.g. RFCs).

@rustrial have you looked at fluxcd/kustomize-controller#426? I made the proposed changes in RFC format and waited almost a month before merging them. We posted the above PR to Slack, we had a blog post on fluxcd.io website to ask for feedback on the RFC, we've asked for feedback on the Flux mailing list... What else should we've done for you to see changes like this?

[gberche-orange]

@stefanprodan I wonder whether there is a typo in #1916 (comment)

Upgrade procedure

To update the Flux Kustomization manifests in Git see the following table:
Change in the new version What you should do
Version is now v1beta2 Change the version: apiVersion: kustomize.toolkit.fluxcd.io/v1beta2

I understand this suggest to change the kustomize yaml api version to apiVersion: kustomize.toolkit.fluxcd.io/v1beta2 whereas

• https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/ still documents apiVersion: kustomize.config.k8s.io/v1beta1
• Increment kustomization.yaml apiVersion? kubernetes-sigs/kustomize#1430 seems to state that apiVersion: kustomize.toolkit.fluxcd.io/v1beta2 is not yet published

[stefanprodan]

@gberche-orange please see: https://fluxcd.io/flux/faq/#are-there-two-kustomization-types

[gberche-orange]

My bad, I had misread the api group, overlooking the fluxcd.io part. Sorry about the noise

[BharathDevopz8]

I want to deploy awx-operator using flux and kustomize. I am having awx-operator helm release file at base with namespace 'x' and in overlays i am having same helmrelease file with namespace 'dev'. will flux deploys 2 awx-operators in both 'x' and 'dev' namespaces what is the ideal way to run my awx-operator in mutliple environments using flux and kustomize. I am newbie please help