The thing that clicked today: "done" has to be a conclusion the program reaches, not a line it prints on the way out, and the only way to trust a fix for that is to make the old code fail the test first.
The harness is the small program that runs the local model on the GPU box: it sends the prompt, hands the model tools (read a file, list a directory, run a command, search), executes what the model asks, and writes the run's result. An earlier diagnosis had found its worst flaw: the final "status: success" was emitted unconditionally at the end of the main function. A real completion, a model that ran out of turns talking, and a crash all looked the same from outside.
Built / shipped
Phase 0, observability only. Instrumentation that had already been validated in a scratch copy during a model benchmark (token counts per turn, which route each tool call took) was merged into production, nothing else changed. A wrapper call replaced by its own inlined body looked like a structural change and was proven not to be.
Phase 1, parser safety and a bounded read. Three defects reproduced on the unmodified code first: a command fence inside illustrative prose was being picked up instead of the real one that followed; a single word in backticks in ordinary prose was being recovered as a command to run; and a read of one real file came back uncapped at 186,230 bytes, enough on its own to overflow the model's context. Fixes: the fenced-command parser rewritten from one regular expression into a line-by-line state machine that requires an explicit shell label on its own line and a bare closing fence; the bare-backtick recovery deleted outright rather than narrowed; and a read cap of 12,000 bytes that truncates only at a valid character boundary, with a marker whose stated byte count is exactly right. 27 tests, run against the scratch candidate and then again, verbatim, against the deployed file. Identical results.
Phase 2, the terminal contract. Against the unmodified code, four false-terminal defects reproduced: running out of turns with tool activity exited 0 and "success"; bailing after one real tool call did too; a crash inside tool execution produced zero artifacts and no terminal event at all; and a failure to write the response file after a good run still exited 0. The patch replaced three flags that had been conflated into one variable, the terminal reason, set at every exit point, including the natural loop exhaustion and a new catch around the whole loop for unexpected exceptions, and mapped centrally to a status and a distinct exit code: success 0, turns exhausted 10, no tool ran 11, model server error 12, worker exception 13, artifact write error 14. Artifact delivery moved before the terminal event, so a write failure on an otherwise successful run downgrades it. Crash diagnostics bounded at a few kilobytes, cut at character boundaries. Eleven cases, each run in its own real subprocess so the exit code is checked at the true process boundary, not by inspecting a variable. Two machines' consumers of the old status were audited first to confirm nothing downstream read the fields being replaced.
Phase 3 and 3.1, guards and a durable suite. Duplicate-call and context guards, caps on the remaining tool outputs generalized into one shared helper, and a versioned acceptance suite of 135 checks stored outside any temporary path, so the whole thing can be re-run against the deployed file at any time. 135 of 135 on the scratch candidate, confirmed red on the pre-patch baseline, 135 of 135 on the deployed file.
Problems & fixes
The deployed canary, with no model at all. Running the patched harness with a turn limit of zero, so it can never call the model, produced exit 10 and a transcript that is exactly "init" then "done, incomplete, turns exhausted," with both artifacts written carrying a failed-status prefix and the run's token, and no network call. That is the behaviour that used to be exit 0 and "success."
The success canary, with the model. One directory listing then the finish call: exit 0, native tool-call route on both turns, real metrics on both turns, terminal "done, success," both artifacts written.
Two self-caught test-harness bugs in Phase 3, zero harness-code blockers. The tests were wrong twice and said so.
A file rewritten in place, on purpose. The live file was written in place rather than replaced and renamed, because the old content was being executed and a running interpreter reads by position. Owner and mode preserved, every function from the earlier phases proven byte-identical by diffing the sliced ranges, not by reading.
Decisions
Prove the defect red on the unmodified code before patching, every phase. A test that only ever passed proves nothing about the fix.
One terminal reason, set at every exit, mapped centrally. Three flags that could disagree were the root of the conflation.
Deliver artifacts before declaring done, so a delivery failure can change the verdict.
Test exit codes at the process boundary, in a subprocess, with the model server stubbed so no tool ever touches the real system during acceptance.
Do not reuse an exit code the launcher already uses for "never started." A sibling machine flagged that one.
Learned
An unconditional success line is worse than no status, because it teaches everything downstream to trust it.
Deleting a recovery path is cleaner than narrowing it. The bare-backtick rule could not be made safe, so it went.
A loop's natural exhaustion is a distinct way to end and deserves its own reason; Python's for-else form encodes it cleanly.
Verify against the deployed file, not the candidate. Running the suite twice, verbatim, is cheap and it is the only run that counts.
Still open / next
The wide read that overflowed the context was one oversized tool result, and the caps now bound it, but a prompt that asks for a large selection still has to be told to page. And the acceptance suite should run as a gate on every future harness change, not as a thing someone remembers.