Homelab

My control-plane machine vanished for twenty hours. The most useful evidence turned out to be a safety mechanism that did not fire.

A multi-day video conversion job hit two unrelated failures in one day, and both had the same shape: one broken precondition outside any file, turned by the script into thousands of fast, confident per-file failures. A circuit breaker, a pinned driver, and a watcher that looks at rates instead of names.

My storage box was showing offline, so I went looking at the storage box. The fault was a network port on a different machine that happened to be the storage box's only route out.

What started as moving two small apps to the control-plane machine became the complete retirement of a third machine as an application host. Dependency mapping first, an explicit 'park that one' that evidence later overturned, a symlink instead of rewriting a hundred thousand paths, 51 GB reclaimed, and two live credentials caught before they replicated.

A session meant to make a backup claim checkable first read the config wrong and nearly 'repaired' a healthy setup, then proved one restore, then opened an alarm about unbounded growth. The follow-up the same day measured the alarm and found two of its load-bearing facts wrong. Ruling: accept.

My notes sync between three machines went through one of them, which made that machine a single point of failure for the other two. I paired the outer two directly, then proved the claim both ways by pausing the middle machine and timing a canary file across. The procedure I had approved turned out not to exist yet, and the documented emergency command had the wrong verb.

Jobs were queuing behind a busy machine for no better reason than habit about which machine owned that kind of work, so I separated who runs a job from where the job lands.

The standby for my agent server was designed as a warm standby, formally ratified, amended five times, and thrown out the same day for an always-on mesh with edge steering. That died too, on cost and on a Pi with 905 MB of RAM. What shipped is a dormant cold standby with its own free tunnel, a structural split-brain fence, and a health endpoint that tells the truth about what it cannot do.

Three database backups written to network storage two weeks earlier were all zero bytes. The cause was an idle timeout on the mount tearing it down mid-write. The obvious retry then failed instantly for a second, unrelated reason, and the worker stopped and asked instead of improvising.

I assumed every machine could reach every other. Checking every direction properly found a quarter of the paths broken, and one repair job reported success while having quietly undone itself.

My entire knowledge base had no undo, and adding one turned out to be a real distributed-systems puzzle because the same files live on four machines at once.

Turned the fleet dashboard into an installable phone app with alerts tiered by urgency, a critical channel that survives the app itself failing, and a dead-man's switch that can announce its own death.

My busiest compute machine was cooking itself under real load, so instead of just turning the work off, I built layers of protection that let it keep running without frying.

I taught my notes to answer questions by meaning, so I can ask what I decided about X and why and land on the right note even without remembering a single word from it.

I write a lot of scattered notes across a lot of projects, and instead of ever organizing them by hand, I built a pipeline that rebuilds and reorganizes the whole thing every night while I sleep.

A lot of the small jobs I was paying a hosted model to do did not need a frontier model at all, so I started running open-weight models on my own GPU box and let the routine work run for free.

I stopped doing my machines' work at the keyboard and built a way to hand a task to whichever machine should run it, let it run on its own, and pick up the result later.

I caused an outage by running a heavy job on the one machine that holds everything together, then spent the night proving my failover plan actually works.

Pulled fleet health and my entire notes vault onto one gated dashboard, then added semantic search over my own work by borrowing an embedding model already running elsewhere on the network.

Every cheap health check told me my storage link was perfectly healthy while it was actually running at a tenth of its speed, and the real work was proving which piece was lying.

Turned the fleet into a full mesh where every machine can reach every other one directly, passwordless, without hand-managed keys, laying the groundwork for running work anywhere in the fleet.

Adding one wireless card to the GPU box renumbered the PCI bus and renamed two interfaces, not one, which took down the isolated storage network's gateway through a chain nothing had advertised. The rebuild made the two radios a primary and hot spare, with a measured failover gap and an idempotent service.

Moved the public tunnel entrance off the busy compute box onto its own dedicated Raspberry Pi with its own uplink, so a reboot of any other machine can no longer take the whole public site down.