Introduction

Lovenet Home Operations Repository

Managed by Flux, Renovate and GitHub Actions 🤖

        

Kubernetes Cluster Information:

         

Overview

This is the configuration for my GitOps homelab Kubernetes cluster. This cluster runs home software services for my residence. It is quite complex and there are a lot of interdependencies but the declarative nature of GitOps allows me to manage this mesh of code. The software services fall into a few primary categories:

• Home Automation (Home Assistant, ESPHome, Node-Red, EMQX, ZWave JS UI, Zigbee2MQTT)
• Home Metering and Monitoring (Weather Station, Power Monitoring, Sensors)
• Home Security (Frigate, Double Take)
• IOT Devices (WLED, Ratgdo)

Core Components

Infrastructure

• CentOS 9 Stream: Kubernetes Node Operating System.
• crun: Container Runtime implemented in C.
• nVIDIA Container Toolkit: Container Runtime for nVIDIA GPUs.

Networking

• cilium: Kubernetes Container Network Interface (CNI).
• cert-manager: Creates SSL certificates for services in my Kubernetes cluster.
• external-dns: Automatically manages DNS records from my cluster in a cloud DNS provider.
• ingress-nginx: Ingress controller to expose HTTP traffic to pods over DNS.
• Cloudflared: Cloudflare tunnel client.

Storage

• Rook-Ceph: Distributed block storage for peristent storage..
• Minio: S3 Compatible Storage Interface.
• Longhorn: Cloud native distributed block storage for Kubernetes.
• NFS: NFS storage.

GitOps

• Flux2: Declarative Cluster GitOps
• actions-runner-controller: Self-hosted Github runners.
• sops: Managed secrets for Kubernetes which are commited to Git.
• Rennovate: Automated Cluster Management.

⚙️  Configuration

• Home Assistant
• Node Red
• ESPHome

⚙️  Hardware

Network

Click to see a high level physical network diagram

☁️ Cloud Dependencies

Noteworthy Documentation

Initialization and Teardown   Github Webhook   Limits and Requests Philosophy   Debugging  

Home-Ops Search

@whazor created this website as a creative way to search Helm Releases across GitHub. You may use it as a means to get ideas on how to configure an applications' Helm values.

Coral Edge TPU

Coral USB

Info

I didn't seem to have to do any udev rules or build/load the apex driver when passing this device through to frigate. I had to do those things for the Mini PCIe Coral, but the way I'm doing it now (look at frigate mount point), it doesn't seem necessary.

USB Resets

Whenever the coral device was attached to the Frigate container it would trigger the following entry in dmesg and Node Feature Discovery could no longer identify that the node had the device. This resulted in the frigate staying in a 'Pending' state until I unplugged and then plugged in the Coral USB again. It was very annoying that if the frigate container ever terminated, I'd have to unplug and then re-plug the USB.

[ +12.269474] usb 2-5: reset SuperSpeed USB device number 22 using xhci_hcd [ +0.012155] usb 2-5: LPM exit latency is zeroed, disabling LPM.

This hack resolved the USB reset issue: https://github.com/blakeblackshear/frigate/issues/2607#issuecomment-2092965042

Coral Mini PCIe

Info

Not currently working. For some reason it doesn't show up in 'lspci' in my Dell R730XD. I wonder if using a more powerful power supply would make a difference.

nVIDIA Tesla P40

nVIDIA GPU ignored on Host (Dell R730xd, CentOS 9), PCI Passthrough to KVM VM, running CentOS 9 as a K8S node running nVIDIA Container Toolkit pods.

Host

Ignore PCI device

1. Apend to GRUB_CMDLINE_LINUX in /etc/default/grub

intel_iommu=on pci-stub.ids=10de:1b38

2. grub2-mkconfig -o /boot/grub2/grub.cfg

This step doesn't seem to be working. I have to manually add the pci-stub directive to the kernel cmdline when the server boots.

3. reboot

PCI Passthrough to VM (via virt-manager)

1. Add Hardware -> PCI Host Device

In the VM

Blacklist nouveau in VM

1. echo "blacklist nouveau" > /etc/modprobe.d/blacklist-nouveau.conf

2. reboot

Install nVIDIA Driver

dnf config-manager --add-repo http://developer.download.nvidia.com/compute/cuda/repos/rhel9/$\(uname -i)/cuda-rhel9.repo

dnf module install nvidia-driver:550-dkms

Install nVIDIA Container Toolkit

curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo | sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo

dnf install nvidia-container-toolkit

nvidia-ctk runtime configure --runtime=crio

Kubernetes

Install nVIDIA Device Plugin

Helm Chart

Configuration with NFD / LocalAI / Ollama / etc

LocalAI

1. make sure runtime is set correctly
2. confirm that localai is running on the nvidia-container-runtime

stable-diffusion

On node install TCMalloc

dnf install -y gperftools gperftools-deve

Other

nVidia HTOP

Improved nvidia-smi command.

nVidia HTOP

Fan Speed

IPMI Fan Script Discussion

Fan Control Script

Initialization

./init/create-cluster.sh (on master)

./init/prepare-cluster.sh (on laptop)

./init/initialize-cluster.sh (on laptop)

ssh root@master1 rm /etc/kubernetes/manifests/kube-vip.yaml (on laptop)

Teardown

./init/destroy-cluster.sh (on laptop)

Github Webhook

kubectl -n flux-system get receivers.notification.toolkit.fluxcd.io generates token URL to be put into github.com -> Settings -> Webhooks -> Payload URL

• Content Type: application/json
• Secret: <token from kubectl -n flux-system describe secrets github-webhook-token>
• SSL: Enable SSL verification
• Which events would you like to trigger this webhook?: Just the push event.
• Active:

Resources: Limits and Requests Philosophy

In short, do set CPU requests, but don't set CPU limits and set the Memory limit to be the same as the Memory requests.

• CPU Guidance
• Limits Guidance

Debugging

• https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/
• https://dnschecker.org
• https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/
• https://github.com/nicolaka/netshoot
• https://www.redhat.com/sysadmin/using-nfsstat-nfsiostat