The thing that clicked today: a proxy timeout is a decision someone made once with one endpoint in mind, and a faster search that returns different results is not an optimization, it is silent corruption.
I submitted a practice exam in Waypoint, my study workspace, and got "502 failed." I clicked submit again and got "attempt has already been submitted." Permanent error, stuck page.
Built / shipped
The logs answered it before the code did. The proxy recorded a 502 on the submit at 18:49:28, followed by two 409s, with a broken-pipe error upstream: the server was still finishing work the proxy had already abandoned. That is also exactly why the retry produced a 409 rather than a duplicate: scoring had completed and been saved before the response was lost. The proxy's upstream timeout had a five-second default with exactly one special case, 100 seconds for the AI coach endpoint, and nobody had asked what else could exceed five seconds.
The profile. On a copy of the production database, a submission with 19 open gaps took 29.01 seconds, effectively all of it in one routine that finds relevant readings for each gap, about 1.5 seconds per gap: 536,785 regular-expression searches and 44,618 full-content scans.
The first optimization, reverted. Replacing the per-window regex searches with precomputed match positions plus binary search came in at 28.70 seconds, fractionally worse, because it surrendered the original's early exit after 24 matches. It also changed word-boundary semantics at window edges, because a slice of text makes the boundary permissive in a way the full string is not, which put exact-output parity at risk for no gain. Reverted rather than kept for looking clever.
What worked, all output-preserving. A pre-filter: the candidate routine requires a minimum number of priority and context terms in the winning window, and a window is a slice of the section, so a section lacking those terms outright can never yield an accepted window; counting present terms with an early-exiting search is far cheaper than locating every match. Regex searches per submit fell from 536,785 to 2,650. A semi-join instead of a join in the full-text search: the join fanned one section into a row per linked objective and then collapsed them by comparing whole section bodies; an IN-subquery never creates the duplicates, verified across all 4,131 question-by-scope pairs in the corpus with zero differences and 2.1 times faster. A memoized term pattern that had been rebuilt and re-escaped on each of roughly 580,000 calls. A cross-scope window cache keyed on content hash, because the three scope passes overlap heavily and the window result depends only on content and terms.
Measured before and after on the same database copy, with the old module loaded beside the new one: 4 gaps 5.81 to 1.05 seconds, 10 gaps 13.86 to 4.78, 19 gaps 26.75 to 11.72. Four gaps at 5.81 seconds against a five-second timeout was precisely my failure.
Differential verification. A harness loaded the pre-change module beside the new one and compared the complete reading list per case: section, snippet, content hash, ordering, retrieval basis. Four hundred and eighty cases, identical.
The durable fixes. Per-route declared timeouts instead of a second special case: a default of five seconds plus a resolver; submit gets 60, the coach keeps 100, verified against the deployed module. And a 409 is not a dead end: since scoring precedes the response, "already submitted" means the answers are saved, so the client now clears the draft and goes to the results, which also means a future timeout degrades to one extra click.
Problems & fixes
My first instinct was wrong. I assumed the coach's special-cased timeout meant the coach ran somewhere else. It does not; the special case was the only endpoint anyone had ever raised above the default, which pointed straight at the defect.
All timing was measured on throwaway copies. The live check was limited to read-only endpoint probes and a direct check of the deployed timeout resolver, because writing a fake attempt into production is forbidden by the project's own rules, and producing fake evidence to prove a fix is worse than the bug.
Decisions
Fix the slowness, not just the timeout. Raising the timeout alone would have moved the cliff.
Only output-preserving optimizations. A faster search with different results is corruption.
Per-route timeouts rather than another special case.
Treat a 409 on submit as proof the answers are saved.
Learned
A proxy default is a decision someone made once. Ask what else can exceed it.
Equivalence is the bar for a performance change. The first attempt was reverted for risking it, and the four that shipped were verified across every case in the corpus.
"Already submitted" was the good news the UI was reporting as an error.
Still open / next
The remaining 11.72 seconds for 19 gaps is still dominated by the same routine; the next win is structural, precomputing per-section term presence at ingest, and it waits for a measurement that says it matters.