The thing that clicked today: a job that fails fast on each file is not failing fast at all when the cause is outside the files, it is converting one outage into a thousand confident errors and eating the manifest while it does it.
A long-running job was converting 24,789 source videos on the GPU box, reading from network storage and writing results back. It had been running for days with a resume cursor, a done-file, and a reboot guard. On one day it broke twice.
Built / shipped
The diagnosis of both bursts. At 06:42 UTC, an unattended driver upgrade landed under the running job. The kernel module and the user-space library no longer matched, the GPU management interface died, and the hardware encoder vanished. Result: 1,385 encode failures at about twelve per minute, each one a real file reported as unconvertible. At 16:16 UTC, the network storage stalled for 180 seconds. The mount was configured to give up rather than wait, so it reported present files as absent. Result: 1,019 "missing source" failures in a burst, and when I sampled 200 of those files directly, 200 of 200 were present. Neither burst lost data: failures never enter the done-file, so a later pass retries them. Both cost wall clock.
The reboot guard that could never pass. After the reboot to recover from the driver upgrade, the scheduled resume spent its full ten-minute window failing its first check and refused to launch, while the storage was healthy the whole time. The check tested whether a directory was a mountpoint, but that directory is a plain folder holding automatic submounts; the real mount is one level down. The test was unfalsifiable in the wrong direction: it would have refused on every boot forever. Fixed to test the real mount, and relaunched by hand; the first genuine conversion landed on the exact file the driver burst had started on.
A circuit breaker. All eight failure sites in the script now route through one helper that counts consecutive failures with no success between them and aborts at 25. A "missing source" additionally re-probes the mount before believing the miss. It shipped as a new file rather than an in-place edit, because the previous version was executing and the shell reads a script by byte position as it runs; editing it in place can change what the live process reads next. Sandbox-tested against a 100-file manifest of nonexistent files: halted at 25 with a distinct exit code, 25 failures recorded, not 100.
The driver pinned. Twenty-six driver packages held, and the trigger confirmed as the daily unattended-upgrade timer, which would have fired again mid-run the next morning.
Problems & fixes
My own watcher missed the second burst. I had armed it on "encode failures at or above five," the previous failure, and encode failures were legitimately zero throughout the storage stall. It reported healthy through 1,019 failures. The miss was filtering for the last failure instead of for failure in general. The replacement watches rates and shapes: failures per minute and consecutive-failure runs, whatever their class.
Three watchdog bugs caught in the sandbox before launch, each of which would have degraded it silently: a field extraction from the driver version file returned a literal word because the open-module banner has different field positions (now matched by shape, not position); a count-or-zero fallback emitted two lines and broke an integer comparison; and the driver alert re-fired on every tick instead of latching.
Estimates were unstable and I over-committed to one. The job's rate depends heavily on the file-size mix ahead of the cursor; a projection from the first minutes is not an estimate.
Decisions
One helper for every failure site, with a consecutive-failure breaker. A precondition failure should stop the job, not consume the manifest.
A new file, never an in-place edit, for a script that is currently executing.
Pin the driver and name the timer that would have unpinned it.
Alert on rates and shapes, not on the name of the last failure.
Learned
Two failures with one shape: a broken precondition outside any file, turned into thousands of fast per-file failures. The fix is the same for both, and it is a breaker, not better error messages.
A guard can be wrong in the safe-looking direction. "Refuse to launch" felt conservative and was in fact an outage on every boot.
A storage mount configured to give up rather than wait turns a server stall into a lie about the filesystem. The re-probe before believing a miss is cheap.
A watcher fitted to the last incident is blind to the next one.
Still open / next
The resume from the corrected cursor re-walks the files the stalled pass already passed, and the 1,019 falsely-missing files now exist and will convert. A separate second pass of roughly fourteen hours remains, and the breaker is now in front of it.