Troubleshooting

My control-plane machine vanished for twenty hours. The most useful evidence turned out to be a safety mechanism that did not fire.

A phone screenshot of an error turned into a day of storage work: an audit of 4,921 pulled photo edits found 3,777 byte-identical to their originals, 108 GB reclaimed, a 3.4 TB library moved with an instant rename, and five separate ways a copy or a delete can lie to you.

A live practice-exam submission returned a 502 and the retry said 'already submitted', leaving a permanent error in front of a button that could never succeed. The proxy had a five-second default nobody had questioned, the submit path took 29 seconds, and the first optimization made it slower. The fixes preserved output exactly or were reverted.

The fleet dashboard kept raising conformance warnings. Genuine state edges were being amplified by four independent defects in how it observed and reported. Then two screenshots from my phone showed the first fix was incomplete.

Loupe got its first real test suite. Writing it found two things: one of the new tests could not fail, and the search feature had been silently broken since a move two days earlier while the service reported healthy the whole time.

The timer that snapshots Loupe's databases reported success while the newest snapshot in the destination was three days old. Two scheduled runs had failed and a manual test run had overwritten the status. Then a real restore drill, which checked four properties and first failed on my own script's assumption.

A multi-day video conversion job hit two unrelated failures in one day, and both had the same shape: one broken precondition outside any file, turned by the script into thousands of fast, confident per-file failures. A circuit breaker, a pinned driver, and a watcher that looks at rates instead of names.

My storage box was showing offline, so I went looking at the storage box. The fault was a network port on a different machine that happened to be the storage box's only route out.

Every mode of my study assistant returned a server error before the model was ever called. The context packet's fixed part had grown past the safety limit on its own, and one unbounded field guaranteed it would keep growing. The fix went into the shared layer three different callers depend on, with caps chosen by measurement and a degrade path before any error.

The service whose only job is to tell me when a device stops checking in was dead. The previous handoff said it had died the day before. It had never worked once: its sandbox was built before its script ran, and it was the one unit in its batch that nobody wired to the failure notifier.

A session meant to make a backup claim checkable first read the config wrong and nearly 'repaired' a healthy setup, then proved one restore, then opened an alarm about unbounded growth. The follow-up the same day measured the alarm and found two of its load-bearing facts wrong. Ruling: accept.

I had encrypted backups of my private records and had never once put one back. Proving it worked also revealed that the older archive contained nothing at all.

The redaction fix from the day before had quietly made my search index 21% bigger. The cause was one line in the wrong order, the fix was two changes that were useless alone, and the proof was a measurement that matched the prediction to the character.

My notes sync between three machines went through one of them, which made that machine a single point of failure for the other two. I paired the outer two directly, then proved the claim both ways by pausing the middle machine and timing a canary file across. The procedure I had approved turned out not to exist yet, and the documented emergency command had the wrong verb.

The free model on my own hardware kept failing halfway through jobs. The cause was not on my side, and the fix was to stop asking the model to write its own proof of work.

Prospect's two AI background workers were quietly overwriting each other's status, so every service restart redid work that was already done. The handoff called it a one-row cosmetic cleanup. Reading the two workers against each other showed something worse, and the fix cost no GPU time.

Three database backups written to network storage two weeks earlier were all zero bytes. The cause was an idle timeout on the mount tearing it down mid-write. The obvious retry then failed instantly for a second, unrelated reason, and the worker stopped and asked instead of improvising.

Hashing every size-collision candidate on my NAS found fifty-two thousand duplicate groups and about 254 GB reclaimable. The interesting part was everything that nearly made the number wrong.

I assumed every machine could reach every other. Checking every direction properly found a quarter of the paths broken, and one repair job reported success while having quietly undone itself.

The harness that runs my local model was reporting success unconditionally, no matter what happened. Hardening it went in four phases, each one proving the defect on the unmodified code before patching, and ended with a single terminal contract that maps every way a run can end to an honest status and exit code.

The control-plane machine had a config file explicitly setting persistent log storage, and its logs were still vanishing. A vendor file with a higher number won by sort order alone, roughly nine days of history had already rotated away, and half the reported symptom turned out to be my own wrong command.

One machine showed permanently critical on the fleet dashboard while every other check against it was green. The probe was judging health on a ping command's exit code, and a ping that succeeds through a relay returns a real answer, a real latency, and a non-zero exit code.

Every cheap health check told me my storage link was perfectly healthy while it was actually running at a tenth of its speed, and the real work was proving which piece was lying.