Monitoring

My control-plane machine vanished for twenty hours. The most useful evidence turned out to be a safety mechanism that did not fire.

The fleet dashboard kept raising conformance warnings. Genuine state edges were being amplified by four independent defects in how it observed and reported. Then two screenshots from my phone showed the first fix was incomplete.

My storage box was showing offline, so I went looking at the storage box. The fault was a network port on a different machine that happened to be the storage box's only route out.

A sweep after the watchdog repair found the fleet's automation registry listed 13 timers on a machine running 16, two of the missing ones backups. It also nearly 'discovered' a hole that did not exist, because a grep over unit files cannot see configuration that arrives through drop-in directories.

The service whose only job is to tell me when a device stops checking in was dead. The previous handoff said it had died the day before. It had never worked once: its sandbox was built before its script ran, and it was the one unit in its batch that nobody wired to the failure notifier.

A standing read-only collector that checks contract parity, machine-to-machine reachability, required services and timers, and required paths across the fleet. Three design choices carry it: it keeps last-known-good evidence, it separates drift from probe failure, and it never repairs anything.

The control-plane machine had a config file explicitly setting persistent log storage, and its logs were still vanishing. A vendor file with a higher number won by sort order alone, roughly nine days of history had already rotated away, and half the reported symptom turned out to be my own wrong command.

One machine showed permanently critical on the fleet dashboard while every other check against it was green. The probe was judging health on a ping command's exit code, and a ping that succeeds through a relay returns a real answer, a real latency, and a non-zero exit code.

Turned the fleet dashboard into an installable phone app with alerts tiered by urgency, a critical channel that survives the app itself failing, and a dead-man's switch that can announce its own death.

My busiest compute machine was cooking itself under real load, so instead of just turning the work off, I built layers of protection that let it keep running without frying.

I caused an outage by running a heavy job on the one machine that holds everything together, then spent the night proving my failover plan actually works.