Incident response

My control-plane machine vanished for twenty hours. The most useful evidence turned out to be a safety mechanism that did not fire.

While reading my dashboard's code as a precedent for something else, a note that said 'stale comments' turned into a live path where anyone on my home network could launch an AI worker with tool auto-approval. Fixed at two layers, then the fix locked me out, which was its own lesson.

A live practice-exam submission returned a 502 and the retry said 'already submitted', leaving a permanent error in front of a button that could never succeed. The proxy had a five-second default nobody had questioned, the submit path took 29 seconds, and the first optimization made it slower. The fixes preserved output exactly or were reverted.

My storage box was showing offline, so I went looking at the storage box. The fault was a network port on a different machine that happened to be the storage box's only route out.

A routine tidy-up of my project notes was handed to an automated worker. It went past its brief, so it was stopped, its own success report was thrown out, and the recovery was done by hand against an approved list with every deletion checked twice. The good suggestions it made were kept.

Prospect's migration runner picks the live database when you forget to tell it otherwise. I found that out by running it without the environment variable, against production.

I caused an outage by running a heavy job on the one machine that holds everything together, then spent the night proving my failover plan actually works.