The thing that clicked today: a security control can be wired into every path and still be a no-op, because "is it hooked up" and "can it match" are different questions, and only the first one is easy to check.
The working-history search I built a few days earlier works by exporting every AI session from every machine, running it through a redaction step, and indexing what comes out so I can search what I actually did. That redaction step had let three real keys through: the admin keys for the sync service on three different machines, each one sitting in plain text in a searchable index of 400,826 chunks.
Built / shipped
The diagnosis, from source. The redaction function really does run on every path into the index: three call sites, all covered, every byte judged clean. The defect was entirely in the pattern list it consults. Every rule in that list keyed on the secret's surroundings: a vendor prefix like sk-, the word Bearer, or a label like password or token sitting right next to a colon or equals sign. Not one rule looked at the value itself. A 32-character random key with no prefix and no adjacent label, which is exactly what these keys are, was invisible to the design by construction. Three keys on three hosts with zero catches was not bad luck. It was the expected output of a context-only matcher meeting a context-free secret.
The fix, in four layers. First and strongest, a process rule: never produce the secret. Any automated job that needs to prove it found a key reports a short fingerprint of it, never the value, and that rule went into the worker contract. Second, literal redaction: the exporter reads each machine's own key from its own config at startup, holds it in memory only, and redacts literal occurrences, zero false positives but only for that machine's secrets. Third, a shape rule for 32-character runs of letters and digits, the general control, with a discriminator that keeps it from eating the fleet's own evidence. Fourth, a modest widening of the label rule to cover <apikey> style and quoted-JSON shapes.
A schema version bump in the same commit as the pattern change, so every exported conversation is regenerated under the new rules automatically (the exporter re-exports anything whose schema does not match) and the fix can never ship without the reprojection, nor the reprojection without the fix.
A measurement budget, registered before measuring. The shape rule had to match at most 0.5% of chunks; the three known fingerprints had to appear at exactly 4, 6 and 1 rows before the fix, as a canary that the measurement was looking at the right thing; the widened label rule got its own 0.25% ceiling. Writing the thresholds down first meant a bad number could not be quietly rationalized afterwards.
Problems & fixes
The shape rule could not catch one of the three keys. The first version required at least one uppercase letter, one lowercase, and one digit, to exclude the hex hashes these transcripts are full of: git commits, checksums, the very fingerprints this work depends on. Measured on the live corpus it was five times better than budget, 388 rows out of 400,826. Then the canary failed: two of the three keys showed up at exactly the expected counts, and the third was absent. Zero rows. Not a miscount. The reason, established without printing the value: that key happened to be 16 uppercase and 16 lowercase letters and no digits at all. A "must contain a digit" clause could never match it, on any corpus, ever. The predicate had been reasoned from what a random key usually looks like, and one of three real keys did not draw a digit. A variant table of five alternatives made the replacement choice mechanical rather than a matter of taste.
The widened label rule would have destroyed the audit trail. Letting a quote sit between the label and the colon made token match "token": "...", which in this fleet is every job-tracking token in every routing file and worker record. 4,399 of roughly 5,100 new matches were exactly that. The shipped version keeps the original rule byte-identical and adds a separate widened one restricted to the password and API-key labels only. About 284 occurrences, 0.07%.
I measured the rule where it does not run. The redaction runs on the projection's input; I measured it against the index, the projection's output, after truncation, after a filter that drops large encoded blobs, after chunking. Everything the pipeline had already discarded was invisible to my measurement and fully visible to the rule. The realized reach was 4.45% of the corpus, ten times my ceiling. Not a security failure, since no prose or evidence was lost, but the new redaction marker had broken the blob filter and the index grew from 1.85 GB to 2.56 GB. Queued as its own item rather than patched in a hurry.
A job that reported done and had abandoned itself. The reprojection worker returned exit 0 and "completed" after running a no-op and ending its turn with a note that it would pause. Steps eight through ten never ran. Caught by checking the actual files, not the status.
Decisions
Rotate the keys only after the redaction is fixed, reprojected, reindexed and verified to zero. Rotating first would have leaked the new keys by the identical path.
Redaction is a blast-radius control on the retrieval path, not a secret-containment control. The raw, unredacted archive still exists on every machine by design, excluded from search, from version control and from every publication surface. That is now a written decision with a threat model, not an unexamined line in a convention.
Pre-register the threshold, then measure. The two failures above were visible only because the budget existed before the numbers did.
Learned
"Is the control hooked up" and "can the control match" are different questions. Review the discriminator, not the plumbing.
A predicate reasoned from the typical case will miss the real case eventually. One of three actual keys had no digit.
Measure a rule where it executes. Measuring at the convenient downstream point gave a number ten times too small.
Two independent mechanisms agreeing is the strongest evidence I have. The final zero was confirmed by a tokenizing scan and by literal substring matching against each machine's own key, over every evidence file and every index.
Still open / next
The rotation itself went through after verification: downtime 180 seconds on the first machine where I waited on a slow path, 2.2 and 2.3 seconds on the other two once the slow step was dropped, each recovery confirmed from a different machine rather than from the host's own report. The index bloat the new marker caused is the next item, and a per-machine census is needed to prove every host's history has been reprojected under the new rules rather than only the ones whose timers happened to fire.