Introduction

Lovenet Home Operations Repository

Production-grade Kubernetes for a household. GitOps with Flux · Automated dependency updates with Renovate · Self-hosted by design

📖 Overview

This is the live configuration for a multi-node Kubernetes cluster that runs a household — home automation, security cameras, media, document management, AI workloads, and the operational tooling required to keep it all up. Every change lands in Git first; Flux reconciles the cluster from there, and Renovate keeps dependencies current via PRs.

The repo is GitOps-strict: applications are declared as HelmRelease resources, secrets are pulled from 1Password through External Secrets Operator, and clusters are mostly identical except for app selection and sizing. Operational quirks, durability tiers, and security defaults live alongside the manifests in .agents/instructions/ so the conventions are enforceable, not folklore.

🗺️ Architecture

flowchart LR
    Dev[👤 Operator] -->|git push| Repo[(📦 GitHub<br/>home-ops)]
    Renovate[🤖 Renovate] -.->|automated PRs| Repo
    Repo -->|reconciles| Flux[⚙️ Flux]
    Flux -->|deploys| Cluster[☸️ Kubernetes<br/>10 nodes · 168 apps]

    Cluster --> Ceph[(🪨 Ceph<br/>block · default durable)]
    Cluster --> LH[(🐂 Longhorn<br/>+ recurring backups)]
    Cluster --> Garage[(🧺 Garage<br/>S3-compatible)]
    Cluster --> NFS[(🗄️ NFS<br/>beast / brain · bulk media)]

    LH -->|weekly + monthly| NFS
    Garage -->|rclone CronJobs| AWS[☁️ AWS S3<br/>Glacier Deep Archive<br/>offsite DR]

    classDef store fill:#1e293b,stroke:#475569,color:#e2e8f0
    class Ceph,LH,Garage,NFS,AWS store

Storage tiers are picked deliberately per workload — see storage-class.instructions.md for the decision tree.

🧰 Stack at a glance

🖥️ Hardware

Off-cluster infrastructure

🌐 Network

Physical topology (click to expand)

Worker nodes attach to iot and sec VLANs via Multus for direct camera and IoT-device reachability. Cilium peers BGP with the upstream router to advertise the LB pool; external ingress flows through Envoy Gateway behind cloudflared.

📦 What's running

🧠 AI agent pipeline

How local AI agents run, get work, ask for human approval, and produce reports — all without putting data in someone else's cloud unless a task genuinely needs it.

flowchart TB
    subgraph Inputs[Inputs]
        AM[AlertManager alerts]
        Op[Operator chat / voice]
        Cron[n8n cron + webhooks]
    end

    subgraph Frontends[Frontends]
        OWUI[Open WebUI]
        HA[Home Assistant<br/>voice + conversation]
        N8N[n8n workflows]
    end

    subgraph Orchestration[Orchestration]
        Holmes[HolmesGPT<br/>alert RCA]
        LG[LangGraph Agents<br/>agent fleet]
        KC[KubeClaw<br/>retiring]
    end

    subgraph Inference[Inference]
        Ollama[Ollama on P40<br/>qwen2.5:7b/14b]
        Claude[Claude API<br/>escalation only]
    end

    subgraph Tools[Tools]
        Gw[MCP Gateway<br/>Authelia-gated JWT]
        Servers[14× MCP servers<br/>HA · Immich · k8s · Grafana · …]
    end

    subgraph Outputs[Outputs]
        Z[Zulip<br/>approvals + chat]
        P[Pushover<br/>high-priority alerts]
        V[(langgraph-vault<br/>drafts + reports)]
        DB[(Postgres CNPG<br/>checkpoints + memory)]
    end

    AM --> Holmes
    Op --> OWUI
    Op --> HA
    Cron --> N8N

    OWUI --> Ollama
    HA --> Ollama
    N8N --> LG

    Holmes --> Ollama
    LG --> Ollama
    LG -.-> Claude
    KC --> Ollama

    Holmes --> N8N
    LG --> Gw
    KC --> Gw
    OWUI --> Gw
    Gw --> Servers

    N8N --> Z
    N8N --> P
    LG --> V
    LG --> DB

Agent fleet (LangGraph)

A single rwlove/langgraph-agents FastAPI service runs the fleet. Each agent is a LangGraph graph with its own persona, tool set, and cost cap. Postgres-checkpointed state lets long-running plans survive restarts.

Pipeline stages

1. Inbox — langgraph-inbox.json workflow ingests requests from chat, AlertManager, or scheduled triggers.
2. Triage — triager classifies and assigns to a specialist agent.
3. Plan — agent drafts an action plan (goals, steps, tool calls, expected cost) into Postgres state.
4. Approval (HITL) — for anything non-trivial, langgraph-approval-post sends a signed Zulip message + Pushover ping with the plan summary; langgraph-approval-receive waits on the reply; langgraph-awaiting-user-sweep chases stuck tasks.
5. Execute — agent runs tool calls through the MCP gateway. Cost caps enforced by langgraph-cost-cap-watcher ($5/task, $10/agent/day, $30/global/day).
6. Report — output written to langgraph-vault (drafts / finals), summarized into the reporter agent's daily Zulip digest (langgraph-daily-digest).

Local-first by design

health-tracker and errand-runner are pinned local-only — they never escalate, even if quality suffers, because the data class isn't suitable for off-site inference.

Voice-to-action: power button → HA Assist → agents → Obsidian

The most common way work enters the fleet — hold the phone's power button, say "inbox <whatever I'm thinking>", and the cluster takes it from there.

flowchart LR
    Btn[📱 Hold power button<br/>Pixel: 'Hold for Assistant'] --> Assist[HA Companion app<br/>set as default assistant]
    Assist -->|audio stream| HA[Home Assistant<br/>Assist pipeline]
    HA --> Whisper[Whisper STT<br/>wyoming-services on P40]
    Whisper --> Sentence[Sentence trigger:<br/>'inbox &#123;content&#125;']
    Sentence --> Ollama[conversation.ollama_voice<br/>qwen3:8b]
    Ollama --> Rest[HA rest_command<br/>POST + Authelia JWT]
    Rest --> Hook[n8n: langgraph-inbox]
    Hook --> LG[langgraph-agents /inbox]
    LG --> Triage[triager classifies]
    Triage -->|capture only| Note[note-maker]
    Triage -->|plan + act| Spec[specialist agent<br/>drafts plan]
    Spec -->|needs input| Zulip[💬 Zulip approval<br/>+ Pushover ping]
    Zulip -->|reply| Receive[approval-receive]
    Receive --> Spec
    Spec --> Done[outcome to vault]
    Note --> Inbox[/vault/inbox/YYYY-MM-DD-…md/]
    Done --> Outputs[/vault/outputs/&#123;drafts,finals&#125;//]
    Inbox --> Couch[(obsidian-couchdb)]
    Outputs --> Couch
    Couch -->|LiveSync| Phone[📱 Obsidian on phone<br/>same vault]

The path

1. Hold power button. Pixel's "Hold for Assistant" gesture is bound to the HA Companion app as the default digital assistant. The Assist UI opens with the mic hot.
2. Speak. Audio streams to the cluster — no on-phone STT. The trigger phrase is inbox <body>; everything after inbox is the note.
3. STT in cluster. The Assist pipeline routes the audio to Whisper (wyoming-services, GPU-accelerated on the P40).
4. Intent + LLM. A sentence trigger matches inbox {content} and hands {content} to conversation.ollama_voice (qwen3:8b on Ollama, tool-calling enabled). The conversation agent's only job here is to confirm the intent and call the rest_command — it does not interpret the content.
5. Auth'd POST. An HA rest_command POSTs to https://langgraph-inbox.${SECRET_DOMAIN}/webhook with { source:"voice", user:"rob", content:"<transcript>" }. The request carries an Authelia client_credentials JWT issued to a dedicated ha-voice-inbox OIDC client — same daily-rotated signing-key machinery the MCP gateway already uses. Envoy's SecurityPolicy validates the JWT against Authelia's JWKS at the gateway.
6. n8n langgraph-inbox. Normalizes the payload and POSTs to /inbox on langgraph-agents.
7. Triager classifies. Research question, household errand, homelab change, property task, or note-to-self — and picks the specialist agent.
8. Capture path → note-maker writes the file to /vault/inbox/YYYY-MM-DD-HHMM-<slug>.md on the langgraph-vault-rw PVC. Single writer, no race with the phone.
9. Plan-and-act path → specialist drafts a plan into Postgres + a draft under /vault/outputs/drafts/. HITL approval via the existing Zulip + Pushover loop when needed (see triggers above).
10. Round-trip to the phone. obsidian-couchdb watches the vault PVC and replicates new files through Self-hosted LiveSync — the note from step 8, plus any drafts/finals from step 9, appear in the Obsidian app on the phone within a sync cycle. Same surface the dictation started on.

The loop closes locally and on one surface: power-button → speak → outcome appears in the vault. Whisper, Ollama, n8n, and the agents all run in the cluster; the only off-site dependency is claude.com if the local fleet escalates a task.

Alert triage (production today)

HolmesGPT is the one agent already running in production:

• AlertManager → HolmesGPT webhook (via alertmanager-holmesgpt-pushover.json) on every firing alert
• HolmesGPT queries Prometheus, Loki, and the cluster directly to build a root-cause hypothesis
• Result posted back as a Pushover message + Zulip thread; n8n sanitizes raw tool-call descriptors out of the agent text before delivery

Current state (2026-05-16)

• HolmesGPT — live, handling cluster alerts daily.
• LangGraph fleet — plumbed but cold (ENABLE_CLAUDE_API: false, no production triggers). Gated on NVIDIA Spark / Ascent GX10 arrival (~2026-05-20), which becomes the primary Ollama backend before the fleet goes hot.
• KubeClaw — running in parallel during the LangGraph transition; scheduled for retirement once LangGraph is validated.

☁️ Cloud dependencies

🛡️ Operational pillars

💾 Tiered storage durability

Four tiers, picked by what the data has to survive — node loss, Ceph loss, cluster loss, or full site loss. Databases get ceph-block + Barman→Garage; irreplaceable state goes to Longhorn with NFS-shipped weekly + monthly backups; S3-shaped workloads use Garage; bulk media rides direct NFS. Full decision tree: .agents/instructions/storage-class.instructions.md.

🔐 Secrets — zero plain-text in Git

All 109 ExternalSecrets resolve through External Secrets Operator from 1Password. Application credentials are templated into ExternalSecret resources and never live in YAML. Cross-namespace mirrors use the reflector pattern when consumer charts hard-code secret names.

🪪 Authentication — single sign-on everywhere

Authelia (with LLDAP) is the OIDC identity provider; per-app oauth2-proxy instances enforce auth at Envoy Gateway. 24 apps sit behind SSO today. The mcp-gateway validates Authelia-issued JWTs with a daily-rotated signing key for MCP tooling.

🔭 Observability — metrics, logs, AI triage

kube-prometheus-stack scrapes everything; Loki ingests pod logs; Grafana stitches the dashboards. AlertManager fans alerts to Pushover and to HolmesGPT, which runs LLM-driven root-cause investigation against the cluster and posts findings back via n8n.

🎮 GPU workloads

A single NVIDIA P40 (24 GB VRAM) on worker8 drives Ollama (local LLM), ComfyUI (image gen), Whisper STT, Immich's CLIP face/pet recognition, and the immich-pet-tagger fork pinned to a P40-compatible PyTorch build. Driver lifecycle is handled by the NVIDIA GPU Operator.

🛟 Disaster recovery

Per-app rclone CronJobs ship Immich originals and Paperless documents — plus their Garage-stored Postgres backups — to encrypted AWS S3 with a 1-day Glacier Deep Archive transition. Recovery procedure is documented at Offsite recovery and was last validated 2026-05-05.

🌪️ Strict GitOps

Every change reaches the cluster through Git. Flux suspends are a deliberate manual signal — paused Kustomizations are not "broken," they're intentional pauses for in-flight maintenance and are documented in conventions, not reverted on sight.

📚 Documentation

The full operator handbook lives in the mdBook site: https://rwlove.github.io/home-ops/.

Frequently referenced pages:

• Cluster rebuild
• Initialization & teardown
• Cluster upgrade
• Power outage recovery
• Limits & requests philosophy
• Debugging playbook
• Offsite recovery
• Immich restore to new CNPG database
• NVIDIA P40 GPU setup
• master1 etcd disk swap
• GitHub webhook

Repo-local conventions (auto-loaded by AI agents from .agents/instructions/):

• Storage class selection · HelmRelease security defaults · ConfigMap layout · Sorting rules · Schema correction · Persona

🙏 Acknowledgements

Inspired by the k8s-at-home community. @whazor maintains the excellent k8s-at-home search — a great way to discover how others configure the same Helm releases.