Failover

My notes sync between three machines went through one of them, which made that machine a single point of failure for the other two. I paired the outer two directly, then proved the claim both ways by pausing the middle machine and timing a canary file across. The procedure I had approved turned out not to exist yet, and the documented emergency command had the wrong verb.

The standby for my agent server was designed as a warm standby, formally ratified, amended five times, and thrown out the same day for an always-on mesh with edge steering. That died too, on cost and on a Pi with 905 MB of RAM. What shipped is a dormant cold standby with its own free tunnel, a structural split-brain fence, and a health endpoint that tells the truth about what it cannot do.

I caused an outage by running a heavy job on the one machine that holds everything together, then spent the night proving my failover plan actually works.

Adding one wireless card to the GPU box renumbered the PCI bus and renamed two interfaces, not one, which took down the isolated storage network's gateway through a chain nothing had advertised. The rebuild made the two radios a primary and hot spare, with a measured failover gap and an idempotent service.