This is a Flux/Gitops managed k8s cluster following the model used by k8s@home. This repository defines the cluster, and flux watches for updates and pushes them.
The cluster is provisioned as Kairos high availability k3s using kube-vip and Calico for simple to deploy cluster networking.
The nodes have a mix of accelerators.
See bootstrap for more background on provisioning of bare metal nodes.
This repository contains a .devcontainer which is the environment used to manage the k8s cluster
from the CLI. The .devcontainer has some default mounts including the private terraform
inventory and .env which is a local directory for local secret storage. More detail on
bootstrapping can be found in bootstrap/env and k8s-gitops-env.yaml performs the secret setup.
The cluster follows best practices for a server naming scheme
including specifying a geography, environment (dev and prod) and a purpose
and serial number per machine (e.g. sto01, cfg01, etc). DNS for machines are
managed outside of the cluster.
Reliable, secure, and discoverable services are provided by the following:
- metallb: A load balancer for bare metal kubernetes.
- ingress-nginx: Used for proxying services through kubernetes ingress, exposing any service through the LoadBalancer with TLS.
- k8s_gateay: DNS server for all relevant ingress services in the cluster. This relies on an existing local dns server outside of the cluster to perform forwwarding.
- cert-manager: Creates TLS certs using LetsEncrypt for each service in the cluster. Uses
dns01on a DNS server managed outside of the cluster.
The key storage components running within the cluster are:
- rook-ceph: Provides persistent volumes, allowing any application to use the external ceph storage cluster.
- volsync: Backup and restore for persistent volumes.
- democratic-csi: For other non-standard volumes (local, nfs, smb, etc)
Updates to the cluster are managed by Renovate and and a handful of github actions. Renovate will either apply updates silently or send PRs to update packages to the latest versions, which are then automatically pushed to the cluster by flux. Renovate has a bit of a learning curve, so here are the pieces i've put together following the patterns of the k8s-at-home folks:
- See Renovate Docs: GitHub app installation for how to enable Renovate on a github repo
- See the Renovate configuration for this cluster which has separate updates schedules for the
devandprodclusters. Thedevcluster is updated silently, while theprodcluster has minor updates applied on weekends. This config is heavily documented given there are numerous Configuration Options that may be hard to piece together. - See Renovate Dashboard for visibility into what Renovate is doing behind the scenes. This is pretty useful if you start making configuration changes.
- See Renovate Helm Releases for a GitHub action that adds the
neccessary annotations to a
HelmReleaseso that renovate knows how to manage it. In other words, renovate-helm-releases doesn't actually do any updating itself, just prep work to make Renovate work. You have to update your renovate configuration with a regexp, which is a simple solution to avoid adding special code in Renovate itself to support this. This runs as a nightly action to opt in any newly addedHelmRelease.