The thing that clicked today: a dashboard that cries wolf is usually not one bug, it is a real edge multiplied by several small observation defects, and you have to fix the observation before you can see the edge.
The fleet dashboard runs a conformance collector every few minutes and shows per-machine, per-check state. It had been noisy for days. I asked why.
Built / shipped
Four defects found. Single-scan notification sensitivity: one bad reading raised an alarm. Unstable routing for one machine's short name, so a check sometimes reached the wrong place. A bounded read of each check's receipt capped at 4 KiB, which truncated valid JSON and produced a parse failure that looked like a state failure. And stamp parsers that rejected the current formatting of the documents they read.
The fixes, each a small piece of alert-quality engineering. Require two consecutive non-OK collector scans before a per-check alarm, while keeping the cache-stale signal immediate, because that one represents loss of the evidence producer itself and should not wait. Raise the receipt ceiling to 64 KiB and never store raw receipt content in the evidence record, so the cap cannot become a storage problem. Bound concurrent SSH handshakes per target to two, and retry once only on transport-level failures, preserving a genuine negative result from the remote side without retry. Parsers updated for the formats actually in use.
Then the screenshot. After the first repair, a photo of the dashboard from my phone still showed critical. That exposed two more defects in how status was projected: one card still read a database path that had been retired, and jobs that had been deliberately stopped, with an explicit end timestamp, were being reclassified as live stalls once their timestamps aged past the staleness threshold. Both fixed with focused regressions for the two screenshot symptoms.
And a second screenshot at 01:16 UTC caught a one-sweep critical state: one machine's SSH banner and every SSH-backed probe to it timed out together, which also made the checks delegated through that machine to a third one unreadable, although the machine was up and directly reachable on the next sweep. Verified that it was not a stale browser view (the cache timestamp said 01:09) and that the machine's SSH daemon showed normal limits and no outage evidence around the time; a transient, correctly handled by the two-scan rule, and recorded as the shape of a transitive probe failure.
Problems & fixes
Reading a receipt with a fixed-size buffer and then parsing it is a failure waiting for a long receipt. The cap was the right idea with the wrong number, and the fix includes not storing the raw content so the number can be raised safely.
"Stopped" and "stalled" are different terminal states that looked identical once enough time passed. A job with an explicit end timestamp is done, however old the timestamp.
Decisions
Debounce per-check alarms to two scans; do not debounce loss of the collector itself.
Bound concurrency and retry only transport failures, once.
Model terminal states explicitly: ended is not stalled.
Never store raw receipt content in evidence.
Learned
Noise is multiplicative. A real edge times a parser bug times a truncation times a flaky route is a dashboard nobody trusts.
A screenshot is evidence. Two of the defects were only visible from the phone after the first fix shipped.
A probe delegated through a machine inherits that machine's reachability. When the middle machine blinks, the far one looks dark too.
Still open / next
The transitive-probe shape deserves its own display state, "unreadable via," distinct from "down," so the next one-sweep blink reads as what it is.