Distributed systems

My notes sync between three machines went through one of them, which made that machine a single point of failure for the other two. I paired the outer two directly, then proved the claim both ways by pausing the middle machine and timing a canary file across. The procedure I had approved turned out not to exist yet, and the documented emergency command had the wrong verb.

Jobs were queuing behind a busy machine for no better reason than habit about which machine owned that kind of work, so I separated who runs a job from where the job lands.

The standby for my agent server was designed as a warm standby, formally ratified, amended five times, and thrown out the same day for an always-on mesh with edge steering. That died too, on cost and on a Pi with 905 MB of RAM. What shipped is a dormant cold standby with its own free tunnel, a structural split-brain fence, and a health endpoint that tells the truth about what it cannot do.

I stopped doing my machines' work at the keyboard and built a way to hand a task to whichever machine should run it, let it run on its own, and pick up the result later.