The thing that clicked today: a prompt budget with one unbounded input is not a budget, it is a countdown, and the right place to fix it is wherever every caller shares the same code, not inside the one caller that happened to fail first.
Waypoint has a study assistant. It answers questions about what I am studying, grounded in my own progress, open gaps, and retrieved passages from the books I have admitted into the library. Every call assembles a context packet for the model, and the packet has a character ceiling as a safety and cost bound.
Built / shipped
The diagnosis. Every mode of the assistant returned a 500 before the model was invoked, with the message that the context had exceeded its safety bound. The packet's fixed floor (current state, progress, the adaptive curriculum, and the list of open gaps, all before any retrieval) was already 22,988 characters against a 24,000 ceiling. The open-gaps field alone was 14,284 characters, 62% of the floor, across eight open gaps. Retrieval was carefully bounded by a result limit and a character limit, but gaps were passed through with only an arbitrary "first ten" and no per-field size bound. So the defect got strictly worse every time a gap opened, permanently. It had been waiting to happen.
Bound it at the data layer, not inside the assistant. The function that builds the packet is shared: the assistant uses it, the tool-server interface that lets an outside AI ask for study context uses it, the HTTP route uses it, and the command-line tool uses it. All four had the identical unbounded-gaps defect. The bounds went into that shared function: a gap limit mirroring the existing retrieval limit, a cap on gap count, a cap on readings per gap, and gap selection switched to most-recent-first so whatever gets truncated is the least relevant material.
Caps chosen from measurement, not from today's count. The cap on gaps is six. Today's count was eight; using eight would have looked bounded while just encoding the current data. Six left between 1,800 and 9,500 characters of margin under the new ceiling across the four measured queries without ever needing to degrade; eight left as little as 351 characters on the heaviest.
Trim what the model never reads. Each gap's recall prompt, lab scaffold and reading snippet capped at 400 characters for the prompt; readings per gap capped at two; fields the model never used (content hashes, ranks, internal slugs, retrieval bookkeeping) dropped, because they were pure byte overhead sent to a model that only ever consumed title, section and snippet. Question-bank material was already excluded and stays excluded.
Degrade before failing. If the assembled prompt still exceeds the ceiling after the static bounds, it now trims the lowest-value content first, oldest gaps, then weakest citations, then excerpt text down to 200 characters, before ever raising. A 500 is reserved for the case where even the fully degraded packet cannot fit, which is roughly 8,800 characters of state that this fix does not touch; verified that only triggers if the ceiling is set below about 10,000, outside any realistic value.
The ceiling raised to 32,000. With gaps genuinely bounded, the floor dropped to roughly 15,000 to 17,000 depending on content, and 24,000 would have stayed tight against real retrieval sizes (measured 4,877 to 12,606 characters). 32,000 is about 8,000 tokens, a small fraction of what the model handles; it is a safety and cost ceiling, not a capability limit.
Problems & fixes
A bug I introduced and caught before it shipped. The first draft of the excerpt-truncation step had a loop recomputing a truncation length that converged to a fixed point one character above its own exit condition: an infinite loop. Replaced with a single bounded truncation per citation, and the degrade step stress-tested against an artificially tiny budget to confirm it converges and falls back to a clear error only below the irreducible floor (measured: 8,803 characters with every gap and citation dropped).
Verification, then the real thing. 157 of 157 tests. All four measured queries reproduced read-only against the production database through the prompt builder directly: all under the ceiling with 1,811 to 9,547 characters of margin, where before all four failed, even with zero retrieval. Service restarted, health 200. Then live end-to-end through the real service with real authentication and a real model call: the first call returned a 503 for the model being unavailable, investigated as a possible second hidden blocker (the service's sandbox restrictions were the suspects), reproduced the exact call under the same restrictions successfully every time, retried the live endpoint twice more, both 200 with correctly grounded answers in about nine seconds. One transient cold-start blip, not a systemic block. The gaps mode correctly still reports the true total of eight open gaps from the untouched state fields even though only six gaps' full detail is sent.
Decisions
Fix the shared function, not the first caller that failed. Four surfaces had the bug; one fix.
Choose caps from margin measurements across real queries, never from the current data's size.
Degrade in value order before raising. An error is the last resort, reserved for the irreducible floor.
Raise the ceiling only after the unbounded input is bounded, and for headroom, not as the fix.
Learned
An unbounded field inside a bounded budget guarantees a future outage. The question was never whether, only which gap would be the one.
The best cap is the one with margin, not the one that matches today. Eight would have passed today and failed next week.
Send the model what it reads. A third of the packet was bookkeeping the model ignored.
A loop that converges one step above its own exit is an easy thing to write and a quick thing to test for.
Still open / next
The shared packet builder now bounds gaps, but the adaptive curriculum and progress blocks are still sent whole, and they are the floor. If they grow the way gaps did, the same analysis applies.