Introduction

Warning

These docs contain information that relates to my setup. They may or may not work for you.

---

# Lovenet Home Operations Repository _Production-grade Kubernetes for a household._ **GitOps** with Flux · **Automated dependency updates** with Renovate · **Self-hosted by design**
[](https://github.com/rwlove/home-ops/issues/10329) [](https://fluxcd.io) [](https://rwlove.github.io/home-ops/) [](https://github.com/rwlove/home-ops/actions/workflows/lychee.yaml)
     

---

📖 Overview

This is the live configuration for a multi-node Kubernetes cluster that runs a household — home automation, security cameras, media, document management, AI workloads, and the operational tooling required to keep it all up. Every change lands in Git first; Flux reconciles the cluster from there, and Renovate keeps dependencies current via PRs.

The repo is GitOps-strict: applications are declared as HelmRelease resources, secrets are pulled from 1Password through External Secrets Operator, and clusters are mostly identical except for app selection and sizing. Operational quirks, durability tiers, and security defaults live alongside the manifests in .agents/instructions/ so the conventions are enforceable, not folklore.

---

🗺️ Architecture

flowchart LR
    Dev[👤 Operator] -->|git push| Repo[(📦 GitHub<br/>home-ops)]
    Renovate[🤖 Renovate] -.->|automated PRs| Repo
    Repo -->|reconciles| Flux[⚙️ Flux]
    Flux -->|deploys| Cluster[☸️ Kubernetes<br/>11 nodes · 188 apps]

    Cluster --> Ceph[(🪨 Ceph<br/>block · default durable)]
    Cluster --> LH[(🐂 Longhorn<br/>+ recurring backups)]
    Cluster --> Garage[(🧺 Garage<br/>S3-compatible)]
    Cluster --> NFS[(🗄️ NFS<br/>beast / brain · bulk media)]

    LH -->|weekly + monthly| NFS
    Garage -->|rclone CronJobs| AWS[☁️ AWS S3<br/>Glacier Deep Archive<br/>offsite DR]

    classDef store fill:#1e293b,stroke:#475569,color:#e2e8f0
    class Ceph,LH,Garage,NFS,AWS store

Storage tiers are picked deliberately per workload — see storage-class.instructions.md for the decision tree.

---

🧰 Stack at a glance

Layer	Tool	Role
OS	CentOS Stream 9 / 10 (+ Ubuntu 24.04 on Spark)	Node operating system
Runtime	cri-o + crun (containerd on Spark)	CRI + OCI runtime; Spark is the lone containerd node
Kubernetes	v1.35.4	Control-plane and node version
GPU	NVIDIA GPU Operator + Container Toolkit	P40 on worker8 (Pascal, 24 GB); GB10 on Spark (Grace-Blackwell, 128 GB unified)
GitOps	Flux2	Declarative cluster reconciliation
Automation	Renovate + GitHub Actions	Dependency PRs, link checks, self-hosted runners
CNI	Cilium (eBPF)	Networking, BGP peering, LoadBalancer pool
Ingress	Envoy Gateway	L7 gateway / HTTPRoute
Service mesh	Istio	mTLS + traffic mgmt for mcp-system
Admission	Kyverno	Namespace-delete blast-radius + audit-mode policies
DNS	external-dns	Cloudflare + bind9 split-horizon
TLS	cert-manager	Let's Encrypt + internal CA
Tunnel	cloudflared	Public ingress without exposing home WAN
AuthN/Z	Authelia + Envoy extAuth	SSO; per-route SecurityPolicy ext-authz gates apps
Secrets	External Secrets Operator + 1Password	116 ExternalSecrets, zero plain-text in Git
VPN	wg-easy	Operator OOB WireGuard access
Storage	Rook-Ceph, Longhorn, Garage, direct NFS	Tiered by durability requirement
Databases	CloudNative-PG, Dragonfly, Qdrant	24 Postgres clusters, KV, vector
Observability	kube-prometheus-stack, Loki, Tempo, Grafana	Metrics, logs, traces, dashboards
Telemetry	OpenTelemetry Collector + Vector	Trace pipeline (→ Tempo) + log shipping (→ Loki)
Images	ZOT	Pull-through registry / local cache

---

🖥️ Hardware

Role	Hostname	Device	CPU	RAM	OS	Storage / Accelerators	Notes
🧠	master1	bare-metal	4	32 GB	CentOS 10	NVMe (Longhorn)	Intel iGPU · RTL-SDR · control plane
🧠	master2	VM on beast	3	10.5 GB	CentOS 9		virtualized control plane
🧠	master3	VM on beast	3	10.5 GB	CentOS 9		virtualized control plane
💪	worker2	ThinkCentre M910x	8	32 GB	CentOS 9	NVMe (Longhorn + Ceph OSD)	ZWA-2 Z-Wave dongle
💪	worker3	ThinkCentre M910x	8	64 GB	CentOS 9	NVMe (Longhorn + Ceph OSD)	Sonoff Zigbee dongle
💪	worker4	ThinkCentre M910x	8	32 GB	CentOS 9	NVMe (Longhorn + Ceph OSD)	Coral USB TPU
💪	worker5	VM on beast	10	21.5 GB	CentOS 9	NVMe (Longhorn + Ceph OSD)
💪	worker6	VM on beast	10	26.5 GB	CentOS 9	NVMe (Longhorn + Ceph OSD)
💪	worker7	VM on beast	10	26.5 GB	CentOS 9	NVMe (Longhorn + Ceph OSD)
🎮	worker8	VM on beast	10	51.5 GB	CentOS 9	NVMe (Longhorn + Ceph OSD)	NVIDIA P40 (24 GB VRAM)
🚀	spark	NVIDIA DGX Spark	20	128 GB	Ubuntu 24.04	NVMe + 8 GPU slots	NVIDIA GB10 (Grace-Blackwell, 128 GB unified); arm64 · containerd outlier

Off-cluster infrastructure

Host	Role
beast	Dell R730xd · iDRAC 8 · RAID6 bulk storage · primary NFS · Longhorn backup target · Garage substrate · VM host
brain	Router/gateway · RAID6 mass_storage · NFS for downloads & TV · OOB SSH on :3231

---

🌐 Network

Physical topology (click to expand)

Network	CIDR	VLAN
Default	192.168.0.0/16	0
IoT	10.10.20.0/24	20
Guest	10.10.30.0/24	30
Security (cameras)	10.10.40.0/24	40
Kubernetes pod subnet (Cilium)	10.42.0.0/16	—
Kubernetes services subnet (Cilium)	10.43.0.0/16	—
Kubernetes LB pool (CiliumLoadBalancerIPPool)	10.45.0.0/24	—

Worker nodes attach to iot and sec VLANs via Multus for direct camera and IoT-device reachability. Cilium peers BGP with the upstream router to advertise the LB pool; external ingress flows through Envoy Gateway behind cloudflared.

---

📦 What's running

🏠 Home Automation — Home Assistant ecosystem, 400+ devices

| App | Purpose | |-----|---------| | **Home Assistant** | Primary orchestrator; 400+ Z-Wave / Zigbee / Matter / ESPHome devices | | **ESPHome** | Build & deploy firmware for DIY sensors | | **EMQX** | MQTT broker | | **Node-RED** | Visual automation flows | | **Zigbee2MQTT** | Zigbee bridge (Sonoff stick on worker3) | | **Z-Wave JS UI** | Z-Wave bridge (ZWA-2 stick on worker1) | | **Matter Server** | Matter protocol bridge | | **Frigate** | NVR + ML camera analysis (7+ cameras, Frigate+ trained model) | | **NetBox** | IPAM / DCIM | | **wyoming-services** | Piper TTS + Whisper STT for voice | | **smtp-relay** | Maddy → Mailgun outbound mail |

🎬 Media & Entertainment — Jellyfin, Immich, Music Assistant, RomM

| App | Purpose | |-----|---------| | **Jellyfin** | Primary media server (read-only metadata) | | **Immich** + **immich-pet-tagger** + **immichkiosk** + **immich-power-tools** | Photo library with ML face/pet recognition, offsite-backed | | **Music Assistant** + **Gonic** | Multi-room music control + Subsonic API | | **RomM** | Retro game library (~10k ROMs) | | **Beets** | Music library tagging | | **av1corrector / videodupfinder / medialyze** | Custom video tooling | | **Theme Park** | Consistent UI theming across apps | | **Batocera Webdashboard Pro** | Retro-gaming console dashboard | | **kodi-playback-watcher** | Bridge for Kodi playback state |

🤖 AI & ML — Local inference, agents, image generation (namespace ai/)

| App | Purpose | |-----|---------| | **Ollama** (P40) | Local LLM serving on the Pascal P40 (≤8b-class models, embeddings) | | **Ollama Spark** | LLM serving on Spark/GB10 (qwen3-next:80b-a3b-instruct-q4_K_M for Open WebUI, bge-m3 embeddings) | | **ComfyUI** | Image generation workflows | | **Khoj** | Personal AI assistant over notes + docs (Authelia extAuth-gated) | | **Paperless-AI** | Auto-tagging for paperless-ngx | | **tei-spark** | Text-embedding-inference reranker (unsuspended 2026-05-21) |

📊 Observability — Prom/Loki/Grafana

| App | Purpose | |-----|---------| | **kube-prometheus-stack** | Prometheus + AlertManager + node-exporter | | **Loki** | Log aggregation | | **Tempo** | Distributed tracing backend (SingleBinary mode) | | **OpenTelemetry Collector** | Trace ingestion pipeline (apps → OTel → Tempo) | | **Vector** | Log shipping (sources → Loki) | | **Grafana** | Dashboards + alerting UI | | **kube-state-metrics / kube-ops-view** | Cluster state & visualization | | **Goldilocks** | VPA-driven resource right-sizing recommendations | | **Kromgo** | Prometheus → Glance dashboard bridge | | **Netdata** | Per-node real-time metrics | | **network-ups-tools (NUT)** | UPS monitoring & graceful shutdown | | **exporters** | Custom Prometheus exporters |

🗄️ Data & Storage — Databases, object storage, vector search

| App | Purpose | |-----|---------| | **CloudNative-PG** | 24 Postgres clusters with WAL archiving to Garage | | **Dragonfly** | Redis-compatible in-memory store | | **Qdrant** | Vector DB for embeddings / RAG | | **pgAdmin** | Postgres admin UI | | **Rook-Ceph** | Distributed block storage (default durable tier) | | **Longhorn** | Block storage with NFS-backed recurring backups | | **Garage** | S3-compatible object storage (DB backups, app S3 workloads) |

🌐 Network, Auth & Platform — Ingress, SSO, GitOps machinery

| App | Purpose | |-----|---------| | **Cilium** | CNI, BGP, LoadBalancer pool | | **Envoy Gateway** | Ingress / HTTPRoute (30 routes) | | **cert-manager** | TLS certificate lifecycle | | **external-dns** | Cloudflare + bind9 record sync | | **cloudflared** | Public tunnel without exposed WAN | | **Authelia** | OIDC identity provider | | **LLDAP** | Lightweight LDAP directory backing Authelia | | **extAuth SecurityPolicies** | per-route Authelia ext-authz gating app SSO | | **wg-easy** | Primary OOB WireGuard access | | **External Secrets Operator** | 1Password-backed secret materialization | | **Flux2** | GitOps reconciler | | **Renovate** | Image & Helm chart update PRs | | **Kuadrant** | MCP server gateway (Authelia-gated JWT) | | **Kyverno** | Admission controller — namespace-delete blast-radius + audit-mode policies | | **actions-runner-controller** | Self-hosted GitHub Actions runners | | **ZOT** | Pull-through registry cache |

🗂️ Documents & Collaboration — Personal knowledge stack + self-hosted tools

| App | Purpose | |-----|---------| | **Paperless-ngx** | Document scanning, OCR, tagging (CNPG-backed, offsite-backed) | | **Obsidian** + **obsidian-couchdb** | Notes sync (CouchDB w/ Cloudflare rate-limiting) | | **Zulip** | Self-hosted team chat | | **Windmill** | Workflow automation; 8 checked-in TypeScript flows under `kubernetes/apps/home/windmill/workflows/` cover the unified paperless→Qdrant+LightRAG fan-out ingest, paperless + LightRAG RAG tombstones, HA smart-home intent drift, Windmill self-failure-watch, and the workaround upstream-watcher. (The two original split ingests — `paperless-rag-ingest` (Qdrant) + `lightrag-rag-ingest` (graph) — were merged into `paperless-rag-fanout` on 2026-07-20; their `.ts` remain and their schedules are paused for rollback. Down from 23 — the 16 langgraph-agents-fleet-specific flows, including the inbox/approval/digest/DLQ/cost-cap/awaiting-user/reviewer-weekly scripts and the Zulip triager webhook, were removed 2026-07-06 along with the fleet.) | | **ntfy** | Self-hosted push notifications (operator approvals via Android tap actions) | | **BentoPDF** | Self-hosted PDF toolkit | | **Kitchenowl** | Shopping lists + recipe / meal management | | **Open WebUI** | Self-hosted LLM frontend; routes chat to Ollama-Spark (default) / Ollama-P40, and pulls in the MCP gateway as its only tool server (the langgraph-agent-as-model registration and the HolmesGPT tool server were both removed 2026-07-06). RAG via bge-m3 + bge-reranker-v2-m3 over Qdrant | | **SearXNG** | Privacy-respecting metasearch engine | | **Glance** | Personal dashboard / start page | | **Atuin** | Encrypted shell-history sync across machines | | **IT-Tools** | Self-hosted developer toolbox | | **MediKeep** | Personal medical records | | **Nametag** | Name tag / badge generator | | **Pump** + **Pump-cv** | Custom personal apps (`rwlove`-built) |

🔌 MCP Servers — 19 Model Context Protocol servers behind an Authelia-gated gateway

| Server | Exposes | |--------|---------| | **mcp-gateway** | Aggregating gateway; Envoy SecurityPolicy validates Authelia-issued JWTs (daily-rotated key) | | **ha-mcp** | Home Assistant entities + service calls | | **esphome-mcp** | ESPHome dashboard: device YAML edit, validate, logs, compile + OTA flash | | **immich-mcp** | Immich library search + asset metadata | | **kubectl-mcp** | Cluster introspection + safe `kubectl` ops | | **grafana-mcp** | Grafana dashboards + Loki/Prom queries | | **prometheus-mcp** | Direct PromQL access | | **paperless-mcp** | Paperless-ngx document search | | **netbox-mcp** | NetBox IPAM / DCIM | | **github-mcp** | GitHub repo + PR ops | | **omada-mcp** | TP-Link Omada controller | | **searxng-mcp** | Privacy search through SearXNG | | **arr-mcp** | Library-search interface to media-pull apps | | **time-mcp** | Time / timezone utilities (`rwlove/time-mcp` native-SHTTP build) | | **chrome-mcp** | Playwright-driven Chromium browser automation for agents | | **memory-mcp** | Cross-agent knowledge graph backed by Postgres + pgvector (bge-m3 1024-dim) | | **cilium-mcp** | Read-only Cilium / Hubble introspection (kubectl-mcp-style, Cilium-scoped) | | **windmill-mcp** | Aggregated Windmill workspace tools (script run, flow trigger) |

---

🧠 AI architecture (overview)

Local-first by default — chat and retrieval today; the automated agent fleet and its cloud-escalation gate were both removed 2026-07-06 (see Current state below). Claude Code (this CLI) is available for interactive, operator-driven use, but there is no automated pipeline that reaches it or the Claude API on its own.

📖 Full chapter: see docs/src/ai_architecture.md for per-app integration paths, RAG pipelines, the historical agent-fleet roster, and the file:line references behind every claim here.

flowchart TB
    subgraph Surfaces[Surfaces]
        OWUI[Open WebUI<br/>chat + RAG]
        Khoj[Khoj<br/>personal AI]
    end

    subgraph Bridges[Windmill bridges<br/>8 TS workflows]
        WPaperless[paperless-rag-ingest/tombstone.ts]
        WLightrag[lightrag-rag-ingest/tombstone.ts]
        WOther[smart-home-intent-drift.ts ·<br/>windmill-failure-watcher.ts ·<br/>workaround-watcher.ts]
    end

    subgraph Inference[Inference]
        OllamaP40[(ollama / P40<br/>qwen2.5:7b · embeddings)]
        OllamaSpark[(ollama-spark / GB10<br/>qwen3-next:80b-a3b-instruct-q4_K_M · bge-m3)]
    end

    subgraph Tools[Tools + retrieval]
        Gw[MCP Gateway<br/>19 servers]
        Q[(Qdrant<br/>vector DB)]
        Mem[(memory-mcp<br/>pgvector KG)]
    end

    subgraph Outputs[Outputs + observability]
        AM[AlertManager]
        Push[Pushover<br/>direct page]
    end

    OWUI -->|chat| OllamaSpark
    OWUI --> Gw --> Mem
    OWUI --> Q
    Khoj --> OllamaP40
    WPaperless --> Q
    WLightrag --> Q
    AM --> Push

There's no agent fleet, no gated Claude API lane, and no OTLP trace producer left in this diagram — all three were langgraph-agents plumbing, removed 2026-07-06 along with the fleet.

Surfaces and bridges

• Open WebUI (collab/) — primary chat UI. Defaults to qwen3-next:80b-a3b-instruct-q4_K_M on Ollama-Spark. RAG runs over Qdrant with bge-m3 embeds + BGE reranker-v2-m3 in-process. Tool server wired in: the MCP gateway (the langgraph-agent-as-model registration and the HolmesGPT tool server are both gone).
• Khoj (ai/) — parallel personal-AI surface for notes + docs. Self-contained: own embedding pipeline (default gte-small, optionally ollama nomic-embed-text), chat via Ollama-P40. Does not consume the MCP gateway.
• HolmesGPT — removed 2026-07-06 (no value delivered). The windmill-investigate route/receiver and the alertmanager-holmesgpt-notify.ts bridge were removed in Stage 1 (critical alerts now go straight to Pushover); the HolmesGPT deployment itself, its RBAC, its Open WebUI tool-server registration, and its Ollama/Authelia wiring were removed in Stage 2. HolmesGPT no longer exists anywhere in the cluster.
• langgraph-agents — removed 2026-07-06. The FastAPI multi-agent runtime, its Postgres checkpoints (postgres-langgraph-checkpoints, deleted), vault PVCs (langgraph-vault/langgraph-vault-rw, deleted), and both public routes (hai.${SECRET_DOMAIN} and hai-web.${SECRET_DOMAIN}) are gone. sync-receiver — an sshd sidecar that existed solely to expose the langgraph-vault PVCs over rsync — was deleted alongside it. See Agent fleet — status today for the historical roster.
• Windmill (home/) — 8 checked-in TypeScript flows remain under kubernetes/apps/home/windmill/workflows/ (down from 23): the unified paperless-rag-fanout ingest (Qdrant + LightRAG from one pull), Paperless + LightRAG RAG tombstones, HA smart-home intent drift, Windmill self-failure-watch, and the workaround upstream-watcher. The two original split ingests (paperless-rag-ingest, lightrag-rag-ingest) were merged into the fan-out 2026-07-20 — retained on disk, schedules paused for rollback. The 16 langgraph-fleet-specific flows (inbox, approval post/receive, daily digest, cost-cap watcher, awaiting-user sweep, DLQ watcher, weekly operator-drift crons, the approval-flow smoke test, and the Zulip triager webhook) were deleted 2026-07-06.
• Langfuse — removed 2026-07-06. Its only consumer (langgraph-agents) was already gone; the keep-dormant-vs-remove question is now resolved as remove.
• memory-mcp (mcp-system/) — cross-agent knowledge graph on postgres-langgraph-memory with pgvector(1024), bge-m3 embeds via Ollama-Spark. Fully unaffected by the decommission — it's memory-mcp's own database, used by Claude Code and Open WebUI. It just lost langgraph-agents as a consumer.

Agent fleet — retired 2026-07-06

The langgraph-agents multi-agent fleet (triager, supervisor, historian, reporter, reviewer, the five operator personas — storage/network/observability/ml/smart-home — plus errand-runner, researcher, homelab-engineer, and eight more wired-but-cold agents) was removed in its entirety. There is no automated agent pipeline in the cluster today. The full historical roster with per-agent role and trigger detail lives in docs/src/ai_architecture.md, preserved as a record of what was built and retired — not as a status board for anything currently running.

Local-first routing tiers

Tier	Backend	When used
1	qwen2.5:7b on Ollama (P40)	Fast / simple chat, embeddings, voice STT/TTS support
2	qwen3-next:80b-a3b-instruct-q4_K_M on Ollama-Spark (GB10)	Default chat inference (Open WebUI)

Removed 2026-07-06: the third tier — gated Claude API escalation via langgraph-agents (ENABLE_CLAUDE_API, cost caps $5/task · $10/agent/day · $30/global/day) — is gone along with the fleet. There is no automated cloud-escalation tier today.

Voice-to-action: retired, known gap

Before 2026-07-06, holding the phone's power button and saying "inbox <whatever I'm thinking>" routed through Whisper STT → an HA rest_command → Windmill → langgraph-agents /inbox, which triaged the note to a specialist agent and synced results to the Obsidian vault. That backend is gone.

Known gap, not yet fixed: the HA voice rest_command (defined in the separate home-assistant-config repo) still exists and still POSTs toward the deleted Windmill webhook — it will silently fail. The full original design (mermaid diagram + step-by-step path) is preserved as a historical record in docs/src/workflow_automation.md.

Alert triage

There is no automated investigation step today. AlertManager fires a critical alert and pages Pushover directly. HolmesGPT used to sit in between (querying Prometheus, Loki, and the cluster to build a root-cause hypothesis before posting back via Windmill) but was removed 2026-07-06 — no value delivered.

Current state (2026-07-06)

• HolmesGPT — removed. No longer deployed anywhere in the cluster.
• LangGraph fleet — removed in its entirety: the FastAPI runtime, postgres-langgraph-checkpoints, the langgraph-vault/langgraph-vault-rw PVCs, both public routes (hai.${SECRET_DOMAIN}, hai-web.${SECRET_DOMAIN}), and sync-receiver. 16 of 23 Windmill workflows deleted with it. postgres-langgraph-memory and memory-mcp are kept — that database is memory-mcp's own backend, unaffected by this decommission. Known gap: the HA voice "inbox …" intent is now broken (see Voice-to-action above).
• claude-runner — retired 2026-05-23, superseded by the langgraph fleet (its two CronJobs graduated into agent workflows inside langgraph-agents). With langgraph-agents now also removed, that functionality is gone twice over — there is no automated Claude Code or Claude API pipeline anywhere in the cluster today. Claude Code use is interactive-only (an operator running the CLI, same as the session that made this edit).
• KubeClaw — retired (memo project_open_issues_cleanup_2026_05_20).

---

☁️ Cloud dependencies

Service	Use	Cost
1Password	Secret backend for External Secrets	~$65 / yr
Cloudflare	Domain, DNS, tunnel, WAF rate-limiting	Free
GitHub	Repo hosting + CI	Free
Mailgun	Outbound mail relay (via Maddy)	Free (Flex)
Pushover	Push notifications for AlertManager + apps	$10 one-time
Frigate+	Trained ML model for Frigate NVR	$50 / yr
AWS S3 Glacier Deep Archive	Offsite DR for Immich + Paperless (objects + DB backups)	~$1–5 / mo (varies)
		~$10–15 / mo

---

🛡️ Operational pillars

💾 Tiered storage durability

Four tiers, picked by what the data has to survive — node loss, Ceph loss, cluster loss, or full site loss. Databases get ceph-block + Barman→Garage; irreplaceable state goes to Longhorn with NFS-shipped weekly + monthly backups; S3-shaped workloads use Garage; bulk media rides direct NFS. Full decision tree: .agents/instructions/storage-class.instructions.md.

🔐 Secrets — zero plain-text in Git

All 116 ExternalSecrets resolve through External Secrets Operator from 1Password. Application credentials are templated into ExternalSecret resources and never live in YAML. Cross-namespace mirrors use the reflector pattern when consumer charts hard-code secret names.

🪪 Authentication — single sign-on everywhere

Authelia (with LLDAP) is the identity provider; per-route Envoy Gateway SecurityPolicy extAuth (Authelia's ext-authz endpoint) enforces auth at the gateway — 26 apps sit behind SSO today, tiered admin vs household via access_control rules (the oauth2-proxy sidecar fleet was retired 2026-07-01, #12767). Native-OIDC apps (immich, paperless, open-webui, …) authenticate against Authelia directly. The mcp-gateway validates Authelia-issued JWTs with a daily-rotated signing key for MCP tooling.

🔭 Observability — metrics, logs, dashboards

kube-prometheus-stack scrapes everything; Loki ingests pod logs (via Vector); Tempo ingests traces (via OpenTelemetry Collector); Grafana stitches the dashboards. AlertManager fans critical alerts straight to Pushover — no automated triage step (the HolmesGPT investigation path was removed 2026-07-06).

🎮 GPU workloads

Two GPUs split the workload:

• NVIDIA P40 on worker8 (Pascal, 24 GB VRAM) — Ollama for ≤8b-class models + embeddings, ComfyUI, Whisper STT, Immich CLIP face/pet recognition, and the immich-pet-tagger fork pinned to a P40-compatible PyTorch build.
• NVIDIA GB10 on Spark (Grace-Blackwell, 128 GB unified) — the larger Ollama deployment serving qwen3-next:80b-a3b-instruct-q4_K_M for Open WebUI, plus bge-m3 embeddings for the cross-agent knowledge graph and Paperless RAG.

Driver lifecycle is handled by the NVIDIA GPU Operator. Spark is the lone containerd node in an otherwise CRI-O cluster; a NodeFeatureRule auto-skips the GPU container-toolkit DaemonSet on CRI-O nodes.

🛟 Disaster recovery

Per-app rclone CronJobs ship Immich originals and Paperless documents — plus their Garage-stored Postgres backups — to encrypted AWS S3 with a 1-day Glacier Deep Archive transition. Recovery procedure is documented at Offsite recovery and was last validated 2026-05-05.

🌪️ Strict GitOps

Every change reaches the cluster through Git. Flux suspends are a deliberate manual signal — paused Kustomizations are not "broken," they're intentional pauses for in-flight maintenance and are documented in conventions, not reverted on sight.

---

📚 Documentation

The full operator handbook lives at https://rwlove.github.io/home-ops/.

Frequently referenced pages:

• AI architecture
• Cluster rebuild
• Initialization & teardown
• Cluster upgrade
• Power outage recovery
• Limits & requests philosophy
• Debugging playbook
• Offsite recovery
• Immich restore to new CNPG database
• NVIDIA P40 GPU setup
• master1 etcd disk swap
• GitHub webhook

Repo-local conventions (auto-loaded by AI agents from .agents/instructions/):

• Storage class selection · HelmRelease security defaults · ConfigMap layout · Sorting rules · Schema correction · Persona

---

🙏 Acknowledgements

Inspired by the k8s-at-home community. @whazor maintains the excellent k8s-at-home search — a great way to discover how others configure the same Helm releases.

_{This repo has been continuously reconciling itself since March 2021.}