In-cluster decryption with sops-age caused a catastrophic failure of my cluster. Please help me figure out how/why.

[2fst4u]

So I've been using flux for many months in my cluster. I have nearly every yaml relevant to my cluster in my private repo, all nicely organised into folders where relevant. Everything was working perfectly.

I was using Longhorn for storage in my cluster until recently. I got fed up with failures it has and decided to go back to old faithful rook now that it supports arm64 natively (yay!).

I restored all the backups I had, rebuilt everything that didn't, got it all running and started work on my next project, putting encrypted secrets into my repo with sops and age. Previously I'd just kept my secrets in a file elsewhere out of my repo for security obviously. But I want to edit them and manage them the same way as everything else in my cluster in vscode. In comes sops and age.

I actually followed TechnoTim's video on the subject on YouTube, and it was super helpful in getting sops with age up and running on my computer. But getting it working in flux is the next step that I spent a while figuring out. I followed the docs (which, no hate, but I feel like there are a lot of unexplained settings in the guide for this subject that you're just expected to understand. That's for a different time) and got to the point of:

Create a kustomization for reconciling the secrets on the cluster:

flux create kustomization my-secrets \
--source=my-secrets \
--path=./clusters/cluster0 \
--prune=true \
--interval=10m \
--decryption-provider=sops \
--decryption-secret=sops-gpg

But I changed "source" for my existing repo I bootstrapped from, and "path" to "clusters/production" in my repo. And "sops-gpg" for "sops-age" obviously. I had put my encrypted secrets right in "clusters/production" where all my other yamls are, not deeper.

It reconciled, looked like everything worked fine, and then after about half an hour I decided to check on it. I just HAPPENED to see an event in the flux-system namespace saying something like "my-secrets created/changed [every single file in my repo listed out]". My heart dropped, I checked my pods, slowly every single one was deleted. Their PVCs, the data on them, every namespace referenced in my repo. All gone. Many hung due to CRD finalizers etc and many needed tidying.

Weirdly, the flux components ALSO got removed, including the flux-system namespace itself. They hung of course, because it was trying to reconcile, and bootstrapping again just made it delete everything again until the kustomizations had been fully deleted. My flux components aren't referenced in my repo other than the default bootstrapping created though.

Everything not referenced in my repo, like metallb which gets spun up directly from kubctl apply -f [gitrepo] stayed put, so it's clear flux caused this (through my fault, not blaming anyone other than myself).

So, the actual question: what did I do wrong? Does this sound like what I did caused the issues? What should I do differently and how should I have added in-cluster decryption without flux thinking everything in my repo is a new object? Would it have been fine if my secrets were in their own path and I reference that? If this is human error, is there something flux should potentially do to prevent this? Is there something missing from the docs that could have made this clearer? I'm coming from a position of no blame, I'm simply wanting to use my failure as a learning opportunity.

Sorry for the novel, I'm just hoping to put in words what I messed up and hopefully get to the bottom of how to prevent it myself and in others. I couldn't find a more appropriate place to write this out and ask this other than directly where the specialists are.

[stefanprodan]

Only the root Flux Kustomization named flux-system should have the cluster path --path=./clusters/cluster0. In the cluster0 dir there shouldn’t be anything else but Flux Kustomizations. Please see the example repo on how to structure your files https://github.com/fluxcd/flux2-kustomize-helm-example

[2fst4u]

Thank god so I'm not going crazy then. Ok, I will do my best to try again without following the way the tutorial says then.

Now, you touched on repo structure before and it's got me confused about that too. In the docs under https://fluxcd.io/flux/installation/ it says:

After running bootstrap you can place Kubernetes YAMLs inside a dir under path e.g. clusters/staging/my-app

Which to me reads as, in my repo under clusters/production, I can put any yamls directly under production. This is what I have been doing until now. Is this correct?

But in https://github.com/fluxcd/flux2-kustomize-helm-example that you linked earlier it has the following:

Repository structure

The Git repository contains the following top directories:

    apps dir contains Helm releases with a custom configuration per cluster
    infrastructure dir contains common infra tools such as ingress-nginx and cert-manager
    clusters dir contains the Flux configuration per cluster

├── apps
│   ├── base
│   ├── production 
│   └── staging
├── infrastructure
│   ├── configs
│   └── controllers
└── clusters
    ├── production
    └── staging

(Note I'm not using helm. Does this matter for this?)
Now in this diagram it shows your apps are outside your cluster directory. Is this a requirement? I haven't been doing this so now I'm confused about how this has been working.

├── apps
│      ├── base
│      ├── production 
│      └── staging
├── infrastructure
│      ├── configs
│      └── controllers
└── clusters
         ├── production
         │      └─# I have all my yamls here
         └── staging

Should they not be here?

[stefanprodan]

Your YAMLs shouldn’t be in the cluster dir if you have more than one cluster. As a general rule, best is to place app definitions outside of the clusters dir, this gives you the flexibility to patch them per environment, e.g. different ingress domain on dev vs production.

[2fst4u]

Ok I've done a lot of reading on that part and I'm even more confused about how you make it work but that's a topic for another day. In any case the answer to my original question, it comes down to the command I used breaking the cluster, correct?

[2fst4u]

Just coming back to this because I feel like I didn't really close this off to the point that the question is fully answered. If I were to try this again, should I be using a different repo? If I follow the docs and put my secrets in a different repo, will this work as expected?

[2fst4u]

@stefanprodan hoping you can shed some light on this. Is there an updated command that I need to use to bootstrap using the same repo? My flux install is getting very out of date and I'd like to update it but I can't if I don't know the right bootstrap command that won't break everything.

[crazyelectron-io]

Sorry for reviving this old thread, but I too have some struggles with the current documentation around secrets decryption.
As already mentioned, the example in the documentation only shows how to use a separate repo for secrets, but not how to modify the repo that is already created during bootstrap. I have manually added a decryption section (with age key) to gotk-sync.yaml

spec:
  decryption:
    provider: sops
    secretRef:
      name: sops-age

but I don't think that is the correct way. How should I add this decryption definition to the cluster?

[Sacro]

@stefanprodan why can't it?

[smaugs]

Also dropping a line to show my interest :) maybe you can clarify that @stefanprodan.

[stefanprodan]

The root kustomization is not meant to reconcile anything else but the Flux manifests and other Kustomizations. The decryption setting will be remove when rerunning bootstrap to upgrade and Flux will be completely stuck. Please see the example repo on how to structure your own repos and avoid this issue: https://github.com/fluxcd/flux2-kustomize-helm-example

[2fst4u]

The problem though, is what do you have to do to get from this incorrectly structures repo to a good one without it breaking anything?

[stefanprodan]

The simplest way is to run flux uninstall --keep-namespace which doesn't touches any deployed workloads. Another way is to disable pruning on all Flux Kustomizations, docs here: https://fluxcd.io/flux/faq/#how-can-i-safely-move-resources-from-one-dir-to-another