The thing that clicked today: an invariant you have inferred from a diagram is not the same as one you have watched hold, and the gap between them is where a standby fails on the night you need it.
My notes are replicated by a sync service between the control-plane machine, the backup machine and the GPU box. The topology had grown so that the first two only talked through the third. Lose the GPU box and the other two partitioned, which mattered because the cold standby's whole premise is that it can read a current copy of the notes when the control plane is gone.
Built / shipped
A direct pairing, approval-gated. Read-only reconnaissance first, then an exact proposed change, then my go. The change was small: add each machine as a device on the other through the sync service's local API, share the notes folder over the new link, verify from both directions, and re-read each machine's config from disk to prove the change persisted. Assertions: device set exactly the old set plus one, no pre-existing device lost, no duplicates, folder type and paused state unchanged. Nothing on the middle machine was modified. The link came up instantly, direct over the home network rather than through a relay, and the three machines now form a full triangle for the notes folder. Loss of any one no longer partitions the other two.
"Connected" is not "converging." The first report said connected and was held rather than accepted. A convergence canary compared the new link's folder state against the existing link on the same machine and showed the same global byte count, about 2.27 GB, over both. That settled it.
The proof, in the direction that matters. Claiming the triangle survives the middle machine is an inference from symmetric folder types and one bidirectional connection. A phase-two test made it evidence: pause the middle machine on one side, write a file with a random nonce on the other, time its arrival, check its hash on both ends, unpause. The canary arrived byte-identical in 6.0 seconds with the middle machine confirmed disconnected at the moment of arrival. The connection between the outer two stayed direct, never relayed, throughout.
The reverse direction, because that is the one the standby actually uses. Promotion mounts the backup machine's copy and has to survive the control plane's loss, so the direction it consumes is control plane to backup, not the one just proven. A reciprocal canary: 10.21 seconds, hash identical on both ends, direct not relayed, exactly one device changed state and everything else stayed unpaused. The invariant is now proven in both directions, and the standby's premise no longer rests on inference.
Problems & fixes
The procedure I had approved did not exist. The handoff said the phase-two procedure "is already written, follow it, do not re-derive it." The section it pointed at was the pairing procedure: its mutation step adds a device and its rollback removes one. There was no pause step and no canary step anywhere in the document; it explicitly deferred the canary to "separately and later" without writing it. I had approved "proceed" on the stated premise that the procedure existed. So the actual sequence was composed in the session and shown to me again before it fired, because the thing being approved had changed.
The documented break-glass command would have failed. The emergency remedy for "a run died and left a machine paused" used the collection verb on an individual device path, which takes a different set of verbs. That line is the exact thing an operator reaches for at the worst possible moment. Corrected before the canary fired.
A verification that could not tell healthy from broken. The persistence check read the paused flag as an attribute on the device's config entry. The service stores it as a child element. On a perfectly healthy config the attribute read came back empty, indistinguishable from an unpause failure. The run produced a false negative, diagnosed it, and self-corrected. Then, because a run that made an error should not certify its own correction, a clean process re-checked from scratch: all eight expected facts agreed, and the paused-false element was explicitly present for both peers rather than merely absent. That clean process also deliberately reproduced the false-negative shape to show it understood the difference.
The reverse-direction run had the mirror-image bug: a check that read the flag correctly but would have produced a plausible-looking pass in one failure case. A false negative that shouts is better than a silent pass; both got recorded.
Decisions
Pair the outer two directly; leave the middle machine untouched.
Prove the invariant in both directions, with a nonce, a hash on both ends, and the middle machine confirmed down at arrival time.
Re-present the procedure when the premise it was approved on turns out to be false, even if the change looks the same.
A process that made a verification error does not certify its own fix; a clean process does.
Learned
Connected, converging, and carrying data under partition are three different claims, and only the last one was the point.
Inferred symmetry is not evidence. The direction the standby actually consumes was the one never tested.
Read the break-glass command before you need it. The wrong verb is cheap to find on a calm day.
A check that cannot distinguish healthy from broken is not a check. Reproduce the failure shape on purpose.
Still open / next
A run that dies mid-procedure can still leave a machine paused; the convention names that failure mode but nothing automatically detects it yet. A detached watchdog that notices a paused peer and alerts is the obvious next piece. And the residual is explicit: a machine that loses its own link partitions only itself.