Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kustomize targetNamespace overrides Namespace metadata.name field #4570

Open
1 task done
maximveksler opened this issue Jan 27, 2024 · 0 comments
Open
1 task done

Comments

@maximveksler
Copy link

Describe the bug

Deletion of one of the kustomize manifests leads to termination on the flux-system namesapce which flux does not manage.

We're deploying flux2 using the helm chart. flux2 and CRD resources are deployed into flux-system namespace. The flux-system namespace termination triggers a chain of events resulting in a broken state cluster.

Chain of events:
  1. apps kustomize.toolkit.fluxcd.io/v1 is deleted.
  2. flux-system namespace termination applied by flux kustomize controller.
  3. flux2 helm release uninstall applied due to namespace termination => No active flux2 controller running
  4. Left over flux2 CRD resources with metadata.finalizers[finalizers.fluxcd.io] are blocking namespace termination.
  5. Reinstall of the flux2 helm chart is not possible since the namesapce is in terminating state.

This results in a cluster that is not operational which requires external intervantion (i.e.flux uninstall followed by flux2 install and CRD redeployment) to allow recovery & reconciliation loops to resume operation.

I belive the root cause is workspace kustomize targetNamespace directove overriding name in the Namespace spec. This can be clearly seen from:

❯ flux tree ks workspace
Kustomization/flux-system/workspace
└── Kustomization/flux-system/apps
    └── Namespace/flux-system

Kustomization/flux-system/apps should not be owning Namespace/flux-system

Steps to reproduce

kubectl create ns flux-system

helm repo add fluxcd-community https://fluxcd-community.github.io/helm-charts

helm install flux2 fluxcd-community/flux2 --version 2.12.2 -n flux-system

cat << EOF | kubectl apply -f-
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: flux2-namespace-doom
  namespace: flux-system
spec:
  interval: 30s
  url: https://github.com/maximveksler/flux2-namespace-doom
  ref:
    branch: main
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: workspace
  namespace: flux-system
spec:
  interval: 30s
  targetNamespace: flux-system
  sourceRef:
    kind: GitRepository
    name: flux2-namespace-doom
  path: "./workspaces/bug"
  prune: true
  timeout: 1m
EOF

kubectl delete kustomizations.kustomize.toolkit.fluxcd.io apps -n flux-system

Now inspect flux-system namespace to see it's in terminating state.

❯ kubectl get ns flux-system
NAME          STATUS        AGE
flux-system   Terminating   10m

Expected behavior

flux-system would not be affected, and apps Kustomisation will be re-provisioned by the workspace Kustomisation.

Screenshots and recordings

No response

OS / Distro

GKE 1.27.5-gke.200

Flux version

v2.2.2

Flux check

N/A

Git provider

No response

Container Registry provider

No response

Additional context

No response

Code of Conduct

  • I agree to follow this project's Code of Conduct
@maximveksler maximveksler changed the title Kustomize targetNamespace overrides Namespace metadata.name field Kustomize targetNamespace overrides Namespace metadata.name field Jan 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant