Introduction
Lovenet Home Operations Repository
Managed by Flux, Renovate and GitHub Actions 🤖
Kubernetes Cluster Information:
Overview
This is the configuration for my GitOps homelab Kubernetes cluster. This cluster runs home software services for my residence. It is quite complex and there are a lot of interdependencies but the declarative nature of GitOps allows me to manage this mesh of code. The software services fall into a few primary categories:
- Home Automation (Home Assistant, ESPHome, Node-Red, EMQX, ZWave JS UI, Zigbee2MQTT)
- Home Metering and Monitoring (Weather Station, Power Monitoring, Sensors)
- Home Security (Frigate, Double Take)
- IOT Devices (WLED, Ratgdo)
Core Components
Infrastructure
- CentOS 9 Stream: Kubernetes Node Operating System.
- crun: Container Runtime implemented in C.
Networking
- cilium: Kubernetes Container Network Interface (CNI).
- cert-manager: Creates SSL certificates for services in my Kubernetes cluster.
- external-dns: Automatically manages DNS records from my cluster in a cloud DNS provider.
- ingress-nginx: Ingress controller to expose HTTP traffic to pods over DNS.
- Cloudflared: Cloudflare tunnel client.
Storage
- Rook-Ceph: Distributed block storage for peristent storage..
- Minio: S3 Compatible Storage Interface.
- Longhorn: Cloud native distributed block storage for Kubernetes.
- NFS: NFS storage.
GitOps
- Flux2: Declarative Cluster GitOps
- actions-runner-controller: Self-hosted Github runners.
- sops: Managed secrets for Kubernetes which are commited to Git.
- Rennovate: Automated Cluster Management.
⚙️ Hardware
Hostname | Device | CPU | RAM | OS | Role | Storage | IOT | Network |
---|---|---|---|---|---|---|---|---|
master1 | Intel NUC7PJYH | 4 | 8 GB | CentOS 9 | k8s Master | |||
master2 | VM on beast | 3 | 8 GB | CentOS 9 | k8s Master | |||
master3 | VM on beast | 3 | 8 GB | CentOS 9 | k8s Master | |||
worker1 | ThinkCentre M910x | 8 | 32 GB | CentOS 9 | k8s Worker | longhorn NVMe | Coral USB | |
worker2 | ThinkCentre M910x | 8 | 32 GB | CentOS 9 | k8s Worker | longhorn NVMe | zstick-7 | |
worker3 | ThinkCentre M910x | 8 | 32 GB | CentOS 9 | k8s Worker | longhorn NVMe, ceph osd | sec-vlan | |
worker4 | ThinkCentre M910x | 8 | 32 GB | CentOS 9 | k8s Worker | longhorn NVMe | sec-vlan | |
worker5 | VM on beast | 10 | 24 GB | CentOS 9 | k8s Worker | longhorn NVMe, ceph osd | ||
worker6 | VM on beast | 10 | 24 GB | CentOS 9 | k8s Worker | longhorn NVMe, ceph osd | skyconnect | |
worker7 | VM on beast | 10 | 24 GB | CentOS 9 | k8s Worker | longhorn NVMe, ceph osd | iot-vlan | |
worker8 | VM on beast | 10 | 24 GB | CentOS 9 | k8s Worker | longhorn NVMe, ceph osd | iot-vlan |
Network
Click to see a high level physical network diagram
Name | CIDR | VLAN | Notes |
---|---|---|---|
Management VLAN | TBD | ||
Default | 192.168.0.0/16 | 0 | |
IOT VLAN | 10.10.20.1/24 | 20 | |
Guest VLAN | 10.10.30.1/24 | 30 | |
Security VLAN | 10.10.40.1/24 | 40 | |
Kubernetes Pod Subnet (Cilium) | 10.42.0.0/16 | N/A | |
Kubernetes Services Subnet (Cilium) | 10.43.0.0/16 | N/A | |
Kubernetes LB Range (CiliumLoadBalancerIPPool) | 10.45.0.1/24 | N/A |
☁️ Cloud Dependencies
Service | Use | Cost |
---|---|---|
1Password | Secrets with External Secrets | ~$65/yr |
Cloudflare | Domain | Free |
GitHub | Hosting this repository and continuous integration/deployments | Free |
Mailgun | Email hosting | Free (Flex Plan) |
Pushover | Kubernetes Alerts and application notifications | $10 (One Time) |
Total: ~$5.50/mo |
Initialization
./init/create-cluster.sh
(on master)
./init/prepare-cluster.sh
(on laptop)
./init/initialize-cluster.sh
(on laptop)
ssh root@master1 rm /etc/kubernetes/manifests/kube-vip.yaml
(on laptop)
Teardown
./init/destroy-cluster.sh
(on laptop)
Debugging
- https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/
- https://dnschecker.org
- https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/
- https://github.com/nicolaka/netshoot
- https://www.redhat.com/sysadmin/using-nfsstat-nfsiostat
Github Webhook
kubectl -n flux-system get receivers.notification.toolkit.fluxcd.io
generates token URL to be put into
github.com -> Settings -> Webhooks -> Payload URL
- Content Type: application/json
- Secret: <token from kubectl -n flux-system describe secrets github-webhook-token>
- SSL: Enable SSL verification
- Which events would you like to trigger this webhook?: Just the push event.
- Active:
Notes
To get metrics-server to work with kubeadm, you need to do the following if it isn't setup with the clusterconfig provided to kubeadm https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#kubelet-serving-certs
- Need to pull the kubeadm configuration into this repository
Resources: Limits and Requests Philosophy
In short, do set CPU requests, but don't set CPU limits and set the Memory limit to be the same as the Memory requests.
@whazor created this website as a creative way to search Helm Releases across GitHub. You may use it as a means to get ideas on how to configure an applications' Helm values.