← All posts

August 5 - teaching a knowledge base to admit what it does not cover

The thing that clicked today: a retrieval system that says "not covered" is only useful if you can tell whether the corpus lacks the concept or your vocabulary for finding it is wrong, and those are different problems with opposite fixes.

I forked the study-library architecture (an official scope document as the permanent spine, books as enrichment, fail-closed source hashing) into a small knowledge base for one real-world domain: a standard residential purchase contract and three general-audience books about it. The interesting change was how it treats what it cannot cover.

Built / shipped

The inverted coverage gate. In the study library, an objective with no teaching material blocks the build. Here that rule is inverted: an objective the books do not cover reports missing with informational severity, and the build succeeds while honestly declaring its holes. Integrity failures still block: a source hash that drifts, a missing official document, an unknown objective code. The distinction is between "the corpus is incomplete," which is a fact worth publishing, and "the inputs are not what they claim to be," which is a stop.

First compile. Sixty-four objectives derived from the official document's paragraphs and sub-paragraphs; a corpus of 110 sections from three books; all four source hashes verified before ingest (the official document itself pinned at 1,830,938 bytes). Result: 34 complete, 6 thin, 24 missing. Status ready, zero blocking findings. A retrieval lexicon, a deterministic map from each objective code to the phrases that locate it in the books, with its own hash pinned into the build.

A second pass over the gaps. Of the eight coverage gaps that a first review had flagged, three turned out to be phrasing defects in the lexicon: constructions like "inspections access" and "escrow demand" that no consumer book would ever write. Re-authored in natural language, they found their sections. The other five survived as genuine gaps after six to eight additional phrasing variants each were mined against the corpus and returned zero hits. One of those five surfaced real on-topic vocabulary the lexicon had been missing entirely (the books discuss the concept under a different name), which became a fill.

Eight further lexicon entries authored for concepts the official document introduces that the books could not have covered. All eight matched zero of 110 sections, which is the expected and correct result: three of the concepts are new in the current edition of the document, so no book from the prior five years could cover them, and the rest are contract boilerplate general books skip. Before authoring, the corpus was mined broadly for each concept's vocabulary, zero hits on every variant, confirming the concepts genuinely do not appear rather than the phrasing being wrong.

A parent rollup. Objectives are hierarchical; a parent's status now derives in two passes: each code's own status from only its own links, then the parent's final status from its family. Complete if every member is complete or thin; partial if some are covered and some are gaps, with the uncovered children listed in the parent's record; a gap if nothing in the family is covered. A sanity gate checks that two specific parents read as partial, not complete, which is what the evidence says.

Problems & fixes

The trap is that a lexicon gap and a coverage gap look identical in a report: both say "missing." The only way to tell them apart is to try to find the concept with other words, enough times and broadly enough that silence means absence. Three of eight flipped; five held. Without that pass, three of the reported gaps would have been lies about the books.

Counterparty material is searchable but not instructional. Documents published by the other side of a transaction (agents, lenders, builders, title companies) got their own role in the registry: they can be searched, but they do not count toward coverage and are never used as teaching sources. A knowledge base that lets a sales brochure teach the contract is not a knowledge base.

Decisions

Invert the gate for a domain corpus: incompleteness is information, integrity failure is a stop.

Mine before authoring. Every new lexicon phrase is tested against the corpus before it is trusted, and a zero is recorded with the variants tried.

Roll parents up from children in two passes, with the uncovered children named, so "partial" points at what to read next.

Counterparty sources are search-only.

Learned

"Not covered" has two causes and opposite fixes. Bad vocabulary is fixed in the lexicon; real absence is fixed by adding a source or accepting the gap.

A gap that survives eight rephrasings is a finding, not a failure. Three of the eight new-concept gaps are new in the current edition of the document, and the corpus being silent about them is correct.

The honest status is the useful one. Twenty-four missing out of sixty-four is a reading list; a build that hid them would have been a build I could not trust.

Still open / next

The five genuine gaps and the eight new-concept gaps want a source, most likely the official document's own explanatory material rather than another consumer book. And the same second-pass discipline should run every time a book is added, because new text can turn a lexicon gap into a coverage hit without anyone noticing.