← All posts

August 8 - deciding which of my own records the search index is allowed to believe

The thing that clicked today: a retrieval index is a statement about what you trust, and the honest version has a quarantine, a default of "not mine until proven," and a human in the loop for the ambiguous middle.

The fleet's memory layer exports every AI session on every machine into a searchable index, so I can ask what I actually did. Two things needed fixing about what that index believed.

Built / shipped

A retrieval gate, and a quarantine. An audit of how all eight knowledge bases preserve worker transcripts found that exports written under older versions of the exporter were not safe enough for normal retrieval: redaction and structure had improved since, and the old exports had not been regenerated. Rather than trust them, the indexer got a gate: exports below the current schema version are quarantined from indexing while their raw bundles, manifests and evidence are preserved untouched. Rebuilt, the transcript collection went from 5,615 documents to 5,206. Database integrity check ok, 56 of 56 indexer tests.

An identity bug, fixed for the future and admitted for the past. One source tool wrote every session under a shared file name, and the exporter had been treating that name as an identity, so exports collided and overwrote each other's forensic output slots. IDs now derive from the stable relative path in the session tree: 25 live sources produce 25 unique IDs. Seven collided cards archived, four surviving manifests marked quarantined with valid pointers. Unique IDs prevent recurrence; they cannot reconstruct source files that no longer exist, and those records stay explicitly quarantined.

The refusal. Older session notes do not follow the current schema: some lack the strategy voice, some the rationale. They will not be bulk-filled. Inventing voice or rationale would be worse than preserving an explicitly incomplete record, so schema enforcement is prospective, measured by the nightly lint, and the lint gained checks for session schema, journal routing, continuity size and artifact manifests. Receipt across eight knowledge bases: zero errors, four size warnings.

The import, without self-publishing. Before deleting conversations from a cloud account, I exported them: 708 conversations and 16 design chats. A small streaming analyzer compared them against the 6,551 manifests already in the memory layer using source IDs, opening and closing fingerprints, ordered message IDs, dialogue hashes and conservative near-content matching. Every review artifact was privacy-safe by construction: identifiers, timestamps, counts, hashes, route proposals and pointers only. No titles, no message text, no attachments.

Routing, measured before trusted. A first 24-record sample of proposed destinations exposed three false positives. The rule was tightened to privilege explicit project, title and opening-line ownership over generated summaries and incidental fleet vocabulary; a second stratified sample passed 31 of 31 and the import-eligible class fell from 275 to 127. The default for anything unassigned is quarantine, not the general project, because publishing private account content across the whole mesh requires an explicit ownership decision.

A canary that caught a silent shape. Three imported design chats projected zero events because their messages use a nested content shape the decoder did not know. Added the decoder, excluded attachment payloads and injected system blocks, revalidated those three by hand.

A human review queue. The 86 medium-confidence records went into a local-only review queue, served only on loopback, holding bounded redacted excerpts, whose decisions are durable but record-only: promotion stays a separate guarded step. I worked through all 86, approved 85 exact destinations, excluded 1. The promotion plan was tied to the exact queue generation and decision-registry hashes; a dry run produced 85 new bundles with no collisions; apply passed layer, hash and redaction verification; a second apply returned 85 unchanged.

Problems & fixes

Final state: 130 records projected into the memory layer, then 85 more after review; 509 held in a local, permission-restricted, non-indexed quarantine; zero ID drift; zero residual secret-shaped strings in evidence or cards; 32 of 32 tool tests; strict lint across eight knowledge bases at zero errors and zero warnings; the excluded record retained; the original index preserved byte-exact for audit.

The tempting shortcut was to route everything unassigned to the general project and sort it out later. "Later" is after it has replicated to every machine and entered the index.

The sample that failed was the most useful step. Three wrong destinations in 24 is an error rate, and an error rate is something you can decide about. A routing rule you have not sampled is a guess.

Decisions

Quarantine rather than trust. Old-schema exports and collided identities are preserved and excluded from retrieval, not deleted and not believed.

Do not fabricate history. Schema is enforced from now on; the past is measured and left honest.

Default-deny for imported content. Unassigned means quarantined. Review queue decisions are record-only; promotion is a separate, verified, idempotent apply.

Sample the classifier before acting on it, and sample again after tightening.

Learned

An index that contains everything is less useful than one that knows what it can vouch for.

An ID that is not unique is not an ID. The collision was silent until forensic outputs started overwriting each other.

A canary on projected events catches silent schema variants that an apply step's success would hide.

The human queue was small, bounded and worth the hour. Eighty-six decisions with excerpts in front of me beat a rule that would have been wrong three times in twenty-four.

Still open / next

The 509 quarantined records can get manual routing later, in bounded classes, if it is ever worth it; they stay out of retrieval until then. And the gate that quarantines old-schema exports is the mechanism; a per-machine census of what has and has not been re-exported is the bookkeeping that makes it complete.