The thing that clicked today: the exit code of a one-shot ping is not a reliability signal for whether a path is direct, and a monitor that is wrong in the "everything is on fire" direction is the one that gets ignored first.
The storage box showed critical on the fleet dashboard for days, while its file serving, its shell access, and every other check against it were fine in the same snapshot. The machine was healthy. The monitor was not.
Built / shipped
The root cause, confirmed live. The probe ran the private network's ping command once and branched health solely on its exit code. A ping that succeeds through a relay returns a real reply with a real latency, 113 milliseconds here, and a non-zero exit code, because the tool also reports that no direct connection was established. So a working path reporting 113 ms was classified unreachable, its latency field dropped on the floor, and because the machine's card rolls up on maximum severity, that one mislabelled row painted the whole machine red.
Two cross-checks nailed it. The ping binary's own format strings confirmed that a direct path never renders as the literal word "direct"; it renders as an address and port. And the network's status command showed the machine as directly connected at the same moment a single-shot ping said no direct connection, which proves a one-shot ping's exit code is simply not a reliable direct-versus-relayed signal.
The patch, scoped to one function. Classify off the actual reply text rather than the exit code: a reply is reachable, with its latency recorded; relayed on the first attempt is healthy, by my ruling, since the relay is a working path; no reply is the failure. Six tests, mocking the command's output with both shapes, written before the patch and passing after.
Problems & fixes
A suspected display artifact was cleared: a screenshot showed a large latency number on the failing card, and no code path attaches a numeric latency to a failed probe, so it was not rendered from this bug. And no test suite existed to have caught any of it; the only test file was an unrelated weekly notification canary.
One consequence predicted rather than discovered later: the alert had been seeded in its alarm state since the watcher started, so the first correct sweep after the fix would fire exactly one recovery notification. Expected, not a bug. Verified in the alert and notification tables after the sweep.
A live proof afterwards: the same ping flipped between relayed and direct within a minute, in real time, and the probe read healthy both ways.
Decisions
Classify from the reply, not the exit code.
A relayed reply is healthy. The relay is a path.
Tests for the probe first, then the patch, and the predicted recovery notification written down before the sweep.
Learned
A monitor wrong in the scary direction is as dangerous as one wrong in the reassuring direction; it trains you to ignore red.
Exit codes encode the tool's opinion, not the thing you asked. Read the output.
Predict a fix's side effect before it fires, so the one recovery notification reads as expected rather than as a new problem.
Still open / next
A direct-versus-relayed distinction is worth showing on the card as information, not as health, and the probe suite that now exists should grow to cover the other probes that had none.