The thing that clicked today: the second design was better than the first, the third was better than the second, and the third was better mostly because of what it did not require.
The agent server is the control plane: the tool server my AI assistants call, the job dispatcher, the search over my notes. It runs on one small always-on machine. A standby for it had been on the list since the control-plane outage in early July.
Built / shipped
Design one, a warm standby, ratified then amended. Replicate the control plane to the backup machine, promote and demote with scripts, a marker in the shared notes to say who is active, a boot-time guard to stop a rejoin, a drill to prove it. It was formally ratified against the post-migration fleet and five amendments were applied. It existed to enforce one rule: never two connectors on the same named tunnel, because a duplicate connector is exactly the split-brain the design was meant to prevent. Its single most dangerous step was copying the primary's tunnel credential to the standby, the project's one irreversible action, and a step that would create the hazard if a standby's connector ever started by accident.
Design two, an always-on mesh with edge steering, hours later. The clarifying question from me was "what do you mean by standby, I thought we were making a mesh," and it was the highest-value message in the arc: we had been executing a shared plan for two sections while meaning different things by it. Shown the difference, I chose the mesh: every node lit, each with its own tunnel, the edge steering traffic. That inverted the problem. The fleet no longer had to guarantee exactly one active node; each node having its own tunnel satisfied the rule structurally, and the whole promote-demote-marker-guard apparatus and the credential copy disappeared. Deleting an apparatus is a stronger result than building one. One constraint had to be put back on my choice: round-robin steering across nodes is not fine, because the dispatcher's worker tools are bound to the machine they run on; the earlier read/write split survived and became the rule for which hostnames may share and which must pin.
Design three, the one that shipped. Verification before spending found that the edge-steering design needed a paid load-balancing subscription, and that the machine it was meant to land on is a Raspberry Pi 3 with 905 MB of RAM that could not carry a full copy of the control plane. I did not want a new monthly cost. So: one dormant, degraded copy of the control plane on the Pi with a separate free tunnel; promotion mounts the backup machine's copy of the notes over a secure-shell filesystem and restores the latest append-only off-host snapshot of the router's database; a synchronized activation marker; and a pre-start fence on the primary that refuses to start while the marker exists, so the fleet cannot end up with two active servers. The cost, stated plainly: semantic search is unavailable during an outage, because the Pi cannot hold the index.
The drill. Router database restored on the Pi and its integrity check ok; primary stopped; marker replicated and a competing primary start returned failure at the fence; signed-in public access to the standby's health endpoint reported mode degraded-cold-standby, notes checks true, semantic index false, and the three semantic capabilities explicitly unavailable; demotion stopped the Pi, removed the marker, restored the primary, verified its deep health, unmounted the notes. Final state: Pi inactive, mount absent, marker absent, tunnel connector active, primary in mode primary with the index true. Test gate 239 of 239.
Problems & fixes
I was not confident about the edge load-balancing product from memory and said so out loud before designing, then verified from the vendor's own documentation. Saying "I'm not confident" is what creates the obligation to go check, and the check is what killed design two.
The notes still disagreed with themselves about what the Pi was for: one file called it a cold spare in its summary and corrected itself lower down. Three such contradictions were listed and left for a follow-up rather than fixed in passing.
Decisions
A cold standby with its own free tunnel, not a mesh with paid steering and not a warm standby with a copied credential.
Split-brain fenced structurally: a replicated marker and a pre-start refusal on the primary, proven in the drill by a competing start that failed.
A degraded mode that says what it cannot do, in the health endpoint, rather than silently missing features.
No paid edge feature, pool, monitor or recurring cost enabled.
Learned
A design that removes an apparatus is usually better than one that adds a guard for it. The mesh's real contribution was the list of things the warm standby required that it did not.
Verify the vendor capability before the design depends on it. A paid subscription and a 905 MB machine each independently ended design two.
The question "what do you mean by X" is worth asking early. Two sections of work had proceeded on two different meanings of standby.
Honest degraded is better than quiet degraded. The health endpoint lists the unavailable capabilities by name.
Still open / next
Promotion is manual by design and depends on someone noticing the primary is gone; the dead-man's switch from the alerting work is what would tell me. And the Pi's hundred-megabit network port disqualifies it from anything storage-heavy, which the notes now say plainly.