Case Study
A knowledge base engine, and the gate that stops it lying
Five domains, 1.3 million words, and a verification pass that caught a regulatory penalty that never existed.
5
Domains
One engine, forked
~1,650
Source-linked pages
About 1.3M words
14
Python tools
Domain-agnostic substrate
9-20
Golden assertions
Per domain, per release
The premise
After the first knowledge base worked, on peptide therapeutics, the obvious next move was to hand-build the second one. I refused, and instead split the system into a domain-agnostic substrate and a domain-specific configuration layer.
Standing up a new vertical then becomes mostly a configuration exercise: rewrite the taxonomy, update the type enums, rewrite one index generator, write fresh evaluation assertions. Four more followed: financial compliance, vertical SaaS diligence, energy storage consulting, and UAE financial free zones.
The substrate
Fourteen Python tools carry no domain vocabulary at all, which is what makes them portable:
Link resolution and strict wikilink checking
Frontmatter validation against a per-type schema
Index generation from the taxonomy
Staleness tracking, so pages surface for review on a cadence
Contradiction scanning across the vault
Claim extraction, and disclaimer injection by page type
One gate, pass or fail
Nothing counts as done because it looks done. A single maintenance command runs index generation, lint and evaluation together, and must exit PASS: 100 percent wikilink resolution, zero frontmatter errors, every golden check green. On the energy storage build that meant 17 of 17 checks and 1,057 of 1,057 links.
Golden-query regression suites
Each domain carries 9 to 20 written assertions pinning load-bearing facts to specific pages. If a bulk edit silently drops or corrupts a fact, a check flips red instead of a reader finding it six months later. It is regression testing applied to knowledge rather than code, and it protects against the failure mode that matters most: quiet corruption that still looks fine.
The pass that automation cannot replace
Every build also gets an adversarial verification pass before sign-off. Its only job is to disbelieve the content.
On the financial compliance build it caught a fabricated regulatory penalty: a specific enforcement figure that read as entirely plausible and had no primary source behind it. It passed lint. It passed frontmatter validation. It would have passed the automated evaluation. Only a pass built to distrust the text found it.
That is the argument for keeping a human-designed adversarial step in any pipeline where a confident sentence can cause harm. Structure validates structure. It does not validate truth.
Retrieval on top
Each knowledge base carries a chat interface: a FastAPI server, a Chroma vector store, and a React front end with clickable citations back into the vault, so an answer can always be traced to its source page.
| Failure found | Why it mattered |
|---|---|
| A markdown sanitiser silently stripped custom URL schemes | Citations opened the app in a new tab instead of the source page |
| Auto-save wrote files without frontmatter | The vault corrupted quietly during demos, and validation only caught it afterwards |
| The system prompt allowed raw file-path citations | Those got written back into the vault as broken links, flipping the next link check red |
| A hardcoded API port | Everything broke the moment the port changed or the app was deployed |
Each of those cost me once. They are written down in a forking playbook, annotated with the build where each one bit, so the next domain does not repeat them. The playbook is the part of this project I am most pleased with.
Questions#
Is this in production with users?
No. These run locally and support my own work and client engagements. I would rather say that than quote usage figures I cannot defend. What I can defend is that the quality system is real and has caught real failures, including ones I introduced myself.
Why golden queries rather than a model-graded evaluation?
Both have a place, but golden assertions are deterministic and cheap, and they fail loudly. For load-bearing facts that must survive every bulk edit, I want a check that cannot be talked round.
What would you do differently?
Audit the rules earlier. On a related research programme I found the mandatory honesty section had been followed in 16 of 19 files and access-date coverage was 39 percent rather than the 100 percent the rule claimed. Both had been true for months before anyone measured. A rule nobody audits is a preference.