Using Nickel for declarative Kubernetes in production

[gkleen]

I'm currently doing contract work with @savau (uniworx.de).
We're standing up lots of infrastructure quite rapidly and would like to use the large amount of leverage fully declarative configuration of… everything gives us wrt. to sysadmin/developer time.

In particular I've done some work on truly declaratively defining the desired state of our (currently one node, but growing) kubernetes cluster here fully integrated into our NixOS machine config flake (so no traditional GitOps).
Highlights wrt. to the pain we're having with that include stuff like line 81 and the manually generated .patch files we're applying to the helm charts upstream (e.g. gitlab) provides (large .tar.gz encoded as base64 in a yaml string, anyone?).

After some private communication with @vkleen I'm now intending to:

• stop using k3s; it probably won't provide any benefits in our NixOS config at the end
• download all required helm charts from nix, evaluate them to kubernetes resource definition yaml at build time
• rewrite the manual resource definitions in nickel and properly template them (for runtime secrets; kinda like this)
• at nix build time extract a manifest of record labels that the merged nickel requires (@vkleen says I can do that with nickel doc, currently)
• define a mapping of record labels (strings) to shell scripts in nix
• at nix build time assert that all record labels extracted from nickel have a corresponding shell script in nix
• write a python script that consumes the manifest from nix and spits out a json record of record labels to stdout outputs of the shell scripts
• at runtime (i.e. PostStart of kubelet.service) run the python script to generate an ephemeral json file
• use the ephemeral json file as input for the templated kubernetes resource definitions and output them to ephemeral json (would be ideal if that could be one yaml file with multiple documents directly; currently plan to do some postprocessing on an exported json array, splitting it into multiple files and recombining into one yaml file with yq)
• pass the ephemeral yaml to KUBECTL_APPLYSET=true kubectl apply --prune --applyset=nixos

[yannham]

Hi @gkleen, that sounds both like an exciting and ambitious pipeline! Feel free to ask any question, or report any issue you have during your experimentation. In particular, I hope the Nickel LSP provides you with some value - if not (completion doesn't work, etc.), reports would greatly help us improve Nickel 🙂

[gkleen]

There has been some movement on this and not everything worked out as planned.

Experience with nickel was quite good and it's staged to go into production soon. I'm explaining how it's integrated below.

We're following the advice of nickel-lang.org and providing nickel-lang-cli to our machine definitions from the stable revision of the nickel flake.
What little of the language we're using currently I learned directly from the user manual.
I'm not currently using the lsp, I use a rather barebones emacs for everything I do :-)

• We're staying on k3s; couldn't get kubernetes applysets to work properly – custom resource definitions cannot be part of the same applyset as the custom resources themselves, breaking any notion of atomicity of deployments.
  Rancher AddOns just work better.
• Downloading and templating helm charts at nix build time works nicely.
  This will allow us to perform more customization on them than the regular helm values.yaml-templating allows in the future and preserves our ability to pin exact chart contents via outputHash.
• Custom resource definitions get defined in nix and written to yaml, or in nickel as a top level record with undefined fields, as @vkleen suggested.
• Due to the current state of the doc subcommand we restrict ourselves to exactly fields of the form parameters.<name> ever being undefined, that's fine with me.
• At nix evaluation time we use the doc subcommand to extract all fields under parameters and assert that we have runtime generators for values for exactly all of them.
• At runtime we use a tiny python script to run all the value generators and collect their output into a yaml file.
  We then use the record merge operator to inject the values into the nickel template and format it to yaml.
• The resulting yaml files are then combined into a rancher addon and applied to the (currently still one node, but now two of them) kubernetes cluster on service start – even pruning seems to work nicely.