Two mini-PCs, one flat network, and a lot of watchdogs.

A two-node Proxmox cluster in a home office: a laptop with its screen off and a workstation with a Xeon, three VMs, two containers, a Raspberry Pi doing DNS, and a fleet of devices that reach all of it by name from anywhere. It streams the movies, watches the porch, runs a local LLM, mirrors my repos, and serves sermons to my phone. The interesting part is what happens when a piece of it breaks.

Home office · 2025–2026 · Proxmox VE · Tailscale · ZFS · Docker · the docs repo

2Proxmox nodes
3 + 2VMs + containers
17services on one VM
4 TBZFS media pool
2.5 GbEbetween the nodes
3git remotes

The map

Top to bottom: the devices I carry, the two overlay networks they use to get in, the edge of the house, the two Proxmox nodes with their guests, and the small boxes that live on the LAN beside them. Every box is clickable. Hover to see what it talks to.

Physical
Virtual machine
Container
Overlay network
Device
Storage

Tap a box for what it is and what it runs. On a phone the map scrolls sideways.

What it is

Two Proxmox nodes in a home office. book5 is a Galaxy Book laptop with its screen switched off. tower is a ThinkStation with sixteen Xeon cores, 78 GB of ECC memory, and a Quadro handed whole to the media VM. A Raspberry Pi 2 does DNS and breaks cluster ties. Everything else is a guest.

Why it exists

To run the things I'd rather not rent: media, cameras, a local LLM, a git mirror, an ERP to learn on, and a network I can break on purpose. And to give the ops habits I use at work a home: one source of truth per fact, a runbook per host, a log that survives the crash.

What it taught me

Most of the value is in the failure modes. A silent hang taught me to make crashes loud. A lightning strike taught me to check which bridge a VM is actually on. A mirror that came home taught me that "offsite" is a fact about geography, not intent.

How I reach it

Every host runs the Tailscale client and gets a name, so the bare alias works from anywhere: a phone on cellular, the Pocket in a café. Three things complicate that, and each has its own path. A few boxes have no Tailscale of their own, so a Raspberry Pi on the LAN advertises the whole subnet as a route. One VM lives behind Mullvad with lockdown on, which walls the tailnet off, so it's reached by jumping through its host. And a second overlay, Twingate, stays scoped to the web UIs that only answer on the LAN.

FROM ANYWHERE Phone on cellular Tailscale app, Termux The Pocket in a café Omarchy, Tailscale client Any device on the tailnet phones, tablets, laptops Tailscale mesh VPN · MagicDNS bare ssh alias, from anywhere no port forwards no VPN server to babysit tailnet ACL tags the Pi gateway and auto-approves its LAN route the primary path since May Direct, by name book5 · tower · omarchy pihole · pi1 · pi-gw1 Through pi-gw1 Pi 5 subnet router: the whole LAN cameras · router · the sermons CT ProxyJump through tower VM 101, Mullvad walls it off VM 111, Home Assistant ssh host LAN addresses one hop Twingate zero-trust · the second overlay second client LAN-only web UIs Home Assistant · Plex · Proxmox connector on book5 THE RULE bare alias = Tailscale -local = the LAN flipped in May 2026; before that, LAN first two overlays on purpose: when one breaks, the other usually still works the VM on Mullvad is the one host with no tailnet

The bare alias is the tailnet, the -local suffix is the LAN. Flipped in May; before that it was the other way round, and every roaming device carried a reverse tunnel.

Where the bytes live

tower owns the disks. VM disks sit on an SSD pool; the media, the camera recordings, the model files, and the whole-machine backups sit on a 4 TB spinning pool that tower exports to VM 101 over NFS across the 2.5 GbE bridge. The VM itself has one root disk and nothing local worth keeping, so a rebuild is a rebuild, not a recovery.

tower · ZFS the storage node rpool · 370 GB SSD the VM disks media-pool · 4 TB HDD media/ movies · series · music frigate/ camera recordings ollama/ model files backups/ the Mac · the Pocket a ZFS snapshot per backup run NFS · 3.2 T over the 2.5 G bridge VM 101 mounts it /mnt/media-pool · /mnt/ollama one root disk, no local media Plex · Jellyfin read the library qBittorrent download → scan → move into place Frigate writes the camera clips Ollama loads models off the pool one disk, one mount per VM: fewer places to look when something fills up

Code lives in three places on purpose: GitHub, a Gitea on VM 101, and the bare mirror on pi1. A weekly sweep pushes to all three.

What happens when tower hangs

In March a kernel upgrade started hanging tower silently every one to three days. No panic, no log line, just a box that stopped answering until someone walked over. The fix was less about preventing the hang than about making sure the next one was loud, short, and left evidence.

A kernel regression 6.17.13 hung tower silently every one to three days Pin the kernel stay on the last good build until a newer one proves itself Make hangs loud soft + hard lockup ⇒ panic a crash you can see Reboot in 30 s panic=30 · pstore keeps the kernel's last words Read it later pstore + book5's copy of the syslog stream and if it hangs anyway book5's tower-watchdog probes tower every three minutes; three silent minutes and it power-cycles tower's smart plug book5's net-health watchdog probes internet, peers, and DNS separately, and restarts networking only on true isolation on VM 101, autoheal restarts an unhealthy Frigate and pings Home Assistant, so I see when it fired pinned since May; the hangs got rare, not extinct. The plug is the last resort.

A hang used to mean a walk to the office. Now it means a panic, a reboot, a dump in pstore, and the last lines already sitting on the other node.

One small thing it serves

The lab's newest job is the least infrastructural: sermons. The Pocket transcribes a recording locally with Whisper and builds a follow-along page with the audio. A nightly job syncs the pages and the mp3 into a container on book5. A Raspberry Pi in front of it terminates TLS and proxies the request, so the page is on my phone anywhere, with nothing exposed to the internet. The kit has its own page.

The Pocket transcribed locally, nothing to the cloud → transcript · page · mp3 nightly sync checksum read-back CT 316 · sermons, on book5 nginx, answers the LAN only audio + follow-along pages LAN pi-gw1 · Caddy TLS · reverse proxy answers the tailnet only https Phone sermons.jadedviber.com on the tailnet, anywhere a recording transcribed in the morning is a page with audio by the evening, on any device, with no port open to the internet

The container was born on tower and moved to book5 two days later, after tower hung twice in one day. Correlation, not proof; the move is the experiment.

2026 so far

  1. The media library moved from book5 to a ZFS pool on tower, exported over NFS. Plex has read from there since.

  2. A new router collapsed two subnets into one flat network. Both nodes moved to 2.5 GbE at the same time.

  3. A kernel upgrade started hanging tower every one to three days. In May the kernel was pinned, lockups were set to panic, and remote syslog plus pstore went in.

  4. The SSH convention flipped: the bare alias is Tailscale now and the LAN is the fallback. pi1 was reflashed to DietPi. VM 101 was consolidated onto one disk.

  5. Lightning. One Wi-Fi node died. The cable reroute revealed VM 101 had been on 1 GbE the whole time; merged onto the 2.5 G bridge for about a quarter more throughput. A week later a second router replaced the Deco.

  6. tower hung once more on the pinned kernel with an empty pstore, so book5 got a watchdog that power-cycles tower's smart plug after three silent minutes.

  7. The PC and pi1 came home from the office. The git mirror is no longer offsite; GitHub and Gitea are the offsite copies now.

  8. pi-gw1, a Raspberry Pi 5, joined as a Tailscale subnet router. The Pocket replaced the Mac as daily driver. A sermons container was born on tower, tower hung twice in two days, and the container moved to book5. On the 23rd a GL.iNet Flint 3 became the router, the third this year, with the Deco demoted to an access point.

Stack

Proxmox VE ZFS NFS Docker Tailscale Twingate Pi-hole + unbound Mullvad WireGuard OpenWrt rsyslog + pstore systemd timers Caddy · nginx Frigate Home Assistant Ollama faster-whisper

Every host has a page in a typed wiki in the homeLab repo, with the same rule the work projects follow: one fact, one home, everything else points. The recovery runbook for a downed host is a Claude skill that walks reachability, web UI, service health, watchdog logs, and kernel forensics in that order, because that is the order I kept doing it in by hand.

Built in conversation

None of this was designed in one sitting. It accreted, one outage at a time, through dialogue with Claude: what should happen when tower stops answering at 2am, which bridge a VM is really on, whether a mirror in the same building counts as a backup. The lab is where those habits got practiced before they went to work.

The philosophy, or what I'm building right now.