Managed by Flux, Renovate and GitHub Actions 🤖
Kubernetes Cluster Information:
This is the configuration for my GitOps homelab Kubernetes cluster. This cluster runs home software services for my residence. It is quite complex and there are a lot of interdependencies but the declarative nature of GitOps allows me to manage this mesh of code. The software services fall into a few primary categories:
- Home Automation (Home Assistant, ESPHome, Node-Red, EMQX, ZWave JS UI, Zigbee2MQTT)
- Home Metering and Monitoring (Weather Station, Power Monitoring, Sensors)
- Home Security (Frigate, Double Take)
- IOT Devices (WLED, Ratgdo)
- CentOS 9 Stream: Kubernetes Node Operating System.
- crun: Container Runtime implemented in C.
- cilium: Kubernetes Container Network Interface (CNI).
- cert-manager: Creates SSL certificates for services in my Kubernetes cluster.
- external-dns: Automatically manages DNS records from my cluster in a cloud DNS provider.
- ingress-nginx: Ingress controller to expose HTTP traffic to pods over DNS.
- Cloudflared: Cloudflare tunnel client.
- Rook-Ceph: Distributed block storage for peristent storage..
- Minio: S3 Compatible Storage Interface.
- Longhorn: Cloud native distributed block storage for Kubernetes.
- NFS: NFS storage.
- Flux2: Declarative Cluster GitOps
- actions-runner-controller: Self-hosted Github runners.
- sops: Managed secrets for Kubernetes which are commited to Git.
- Rennovate: Automated Cluster Management.
| Hostname | Device | CPU | RAM | OS | Role | Storage | IOT | Network |
|---|---|---|---|---|---|---|---|---|
| master1 | Intel NUC7PJYH | 4 | 8 GB | CentOS 9 | k8s Master | |||
| master2 | VM on beast | 3 | 8 GB | CentOS 9 | k8s Master | |||
| master3 | VM on beast | 3 | 8 GB | CentOS 9 | k8s Master | |||
| worker1 | ThinkCentre M910x | 8 | 32 GB | CentOS 9 | k8s Worker | longhorn NVMe | Coral USB | |
| worker2 | ThinkCentre M910x | 8 | 32 GB | CentOS 9 | k8s Worker | longhorn NVMe | zstick-7 | |
| worker3 | ThinkCentre M910x | 8 | 32 GB | CentOS 9 | k8s Worker | longhorn NVMe, ceph osd | sec-vlan | |
| worker4 | ThinkCentre M910x | 8 | 32 GB | CentOS 9 | k8s Worker | longhorn NVMe | sec-vlan | |
| worker5 | VM on beast | 10 | 24 GB | CentOS 9 | k8s Worker | longhorn NVMe, ceph osd | ||
| worker6 | VM on beast | 10 | 24 GB | CentOS 9 | k8s Worker | longhorn NVMe, ceph osd | skyconnect | |
| worker7 | VM on beast | 10 | 24 GB | CentOS 9 | k8s Worker | longhorn NVMe, ceph osd | iot-vlan | |
| worker8 | VM on beast | 10 | 24 GB | CentOS 9 | k8s Worker | longhorn NVMe, ceph osd | iot-vlan |
Click to see a high level physical network diagram
| Name | CIDR | VLAN | Notes |
|---|---|---|---|
| Management VLAN | TBD | ||
| Default | 192.168.0.0/16 |
0 | |
| IOT VLAN | 10.10.20.1/24 |
20 | |
| Guest VLAN | 10.10.30.1/24 |
30 | |
| Security VLAN | 10.10.40.1/24 |
40 | |
| Kubernetes Pod Subnet (Cilium) | 10.42.0.0/16 |
N/A | |
| Kubernetes Services Subnet (Cilium) | 10.43.0.0/16 |
N/A | |
| Kubernetes LB Range (CiliumLoadBalancerIPPool) | 10.45.0.1/24 |
N/A |
| Service | Use | Cost |
|---|---|---|
| 1Password | Secrets with External Secrets | ~$65/yr |
| Cloudflare | Domain | Free |
| GitHub | Hosting this repository and continuous integration/deployments | Free |
| Mailgun | Email hosting | Free (Flex Plan) |
| Pushover | Kubernetes Alerts and application notifications | $10 (One Time) |
| Total: ~$5.50/mo |
./init/create-cluster.sh (on master)
./init/prepare-cluster.sh (on laptop)
./init/initialize-cluster.sh (on laptop)
ssh root@master1 rm /etc/kubernetes/manifests/kube-vip.yaml (on laptop)
./init/destroy-cluster.sh (on laptop)
- https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/
- https://dnschecker.org
- https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/
- https://github.com/nicolaka/netshoot
- https://www.redhat.com/sysadmin/using-nfsstat-nfsiostat
kubectl -n flux-system get receivers.notification.toolkit.fluxcd.io generates token URL to be put into
github.com -> Settings -> Webhooks -> Payload URL
- Content Type: application/json
- Secret: <token from kubectl -n flux-system describe secrets github-webhook-token>
- SSL: Enable SSL verification
- Which events would you like to trigger this webhook?: Just the push event.
- Active:
To get metrics-server to work with kubeadm, you need to do the following if it isn't setup with the clusterconfig provided to kubeadm https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#kubelet-serving-certs
- Need to pull the kubeadm configuration into this repository
In short, do set CPU requests, but don't set CPU limits and set the Memory limit to be the same as the Memory requests.
@whazor created this website as a creative way to search Helm Releases across GitHub. You may use it as a means to get ideas on how to configure an applications' Helm values.