The thing that clicked today: the local model was not failing at reasoning, it was failing at the plumbing of tool calls, and no amount of prompt tuning would have shown me that. Reading its transcripts did.
The fleet runs a local open-weight model on the GPU box for deterministic work: bounded reads, checks, inventories, the jobs not worth paying a cloud model for. It had been flaky, and the leading guess was that the harness around it was at fault. Before doing anything, a read-only look at its actual transcripts.
Built / shipped
The preflight finding. Across roughly ninety tool invocations in two transcripts, the incumbent model essentially never used the model server's native tool-call field. Every call arrived as a fenced JSON block inside plain text, caught only by the harness's text-fallback parser, which has no schema validation. That is a plausible mechanism for silently dropping required fields from a deliverable, and it explained an earlier report that had omitted its evidence table. Worse, under a controlled sample it frequently emitted its entire multi-step plan, including the final report text, as several JSON blobs in one completion, before any tool in that blob had run. The harness still executed every call for real, but the model's report reflected what it expected the results to say.
An isolated A/B of the real harness. The production harness was copied into a scratch directory, never edited live, and the scratch copy differed by two functional lines (a context-size parameter) plus additive diagnostic events; system prompt, tool schema, executor, parser, temperature, timeouts and turn limits byte-identical, and the production file's hash re-verified unchanged afterward. Six deterministic, assertion-scored tests: a missing-file read that must report the right error; three-file attribution; verbatim reproduction of a file; a read-then-search-then-write ordering; recovering after a missing-file error; and a required report structure. The compatibility test ran first as a gate, then the full suite.
The result. The candidate won 5 of 6 to the incumbent's 2 of 6, used native tool calls on 18 of 18 completions (incumbent: 0 of 11), and ran about twice as fast, both entirely on the GPU with no spill to CPU. Both failed the verbatim-reproduction test, by different defects: one dropped a line's indentation, the other wrote literal escape sequences instead of the characters.
The promotion gate, then the swap. A 22-trial soak at the production context size passed 21 of 22 with 75 of 75 native tool calls, zero malformed calls, zero fabricated facts, and a steady speed band. The one failure was the same verbatim test, not a tool or server fault. The production selector turned out to be a single constant in one file, confirmed by tracing the whole call path; the swap was one line, proven to reach the actual launch command by reproducing the launcher's own template substitution against a scratch copy. A fresh smoke run afterwards showed the new model name in the run's status and the old one absent.
Problems & fixes
The soak that died at 2 of 22, honestly. The first soak exited 0 with no response file. The build agent had backgrounded the 22-trial runner and scheduled itself a wake-up to collect results later, but it was running as a one-shot headless invocation with no session to wake; its turn ended normally, the process exited 0, and the backgrounded soak was killed at teardown in the middle of trial three. Verdict recorded as inconclusive at a 9% sample, not as a pass. The two trials that did complete tracked the A/B numbers closely at the larger context, which was worth knowing and not enough to promote on.
The re-run was synchronous. Twenty-two trials, about 213 seconds each, in the foreground, with the aggregate written by the process that ran them.
A delivery quirk found by the cutover smoke. The first live smoke under the new model ran out of turns without writing its response, because the new model handled the end-of-run footer differently. Diagnosed from the 49-line transcript, fixed so the final tool call owns durable delivery, re-smoked clean.
Decisions
Diagnose from transcripts before benchmarking. The native-versus-fallback tool-call finding was the actual question, and it came from reading, not from running.
A/B in an isolated copy of the real harness with a two-line diff, so the comparison is of the models and nothing else.
Gate on a soak at production context size, and do not promote on an inconclusive run, however promising its two trials look.
One edit, proven to reach the launch command, with a backup and a diff that shows it is the only changed line.
Learned
A model can be "unreliable" because of how it formats tool calls, not because of how it thinks. The fallback parser was doing all the work and had no way to tell me.
Scored tests beat impressions. Six small assertions settled a choice that had been argued by feel for weeks.
A backgrounded process in a one-shot runner dies when the turn ends. Exit 0 and no output is a harness-usage bug, not a pass.
Restore the baseline after measuring. The incumbent model was reloaded after the A/B so the environment was exactly as found until the deliberate cutover.
Still open / next
Both models fail byte-exact reproduction of a file in different ways, so the harness should not rely on a model to copy text verbatim; it should read the bytes itself. And the two stale docstrings that still name the old model are cosmetic, listed, not fixed.