Trent Milam
← Back to work

rag-reliability — Twelve Silent-Failure Gates for RAG

Personal — RAG & Retrieval Systems · 2026

Twelve small, offline, deterministic tools — one per RAG pipeline stage — that turn silent retrieval failures into loud, named signals. Each ships a red/green self-test that reproduces a real failure, catches it, and clears a clean case so the catch isn't rigged.

12/12
gates passing their red/green self-test measured run_all_evals.py; CI re-runs it on every push across Python 3.10–3.12
6×
SyncGate — fewer sources reprocessed vs. reprocess-everything measured
3
other repos here reuse these gates as a shared library measured linkgraph, activerag, chain-rag
  • RAG
  • Reliability
  • Verification
  • Testing
  • Python
View on GitHub

Context

A RAG pipeline is a chain — chunk a document, embed the pieces, index them, retrieve the right ones, trace an answer back to its source. Any stage can fail quietly and corrupt the final answer without ever raising an error. This toolkit makes each of those failures loud.

What I built

Twelve independent tools, one per pipeline stage — embedding-integrity certificates, a citation-provenance gate, a per-element ingestion conservation law, a GraphRAG structural linter, a hardware-aware cost governor, a PII retrievability audit, and more. Each one reproduces a real failure, catches it, and also passes a clean case, so the catch is proven both ways instead of asserted.

Why it matters

run_all_evals.py runs every tool’s self-test in one command, and CI re-runs it on every push across Python 3.10–3.12 — so 12/12 stays continuously verified, not a point-in-time claim. RAG tooling is a crowded field, so each tool claims one narrow idea and cites prior art for the rest; the value is the combination, the packaging, and a rigorous demo. Three of the other repos here — linkgraph, activerag, and chain-rag — reuse these gates as a shared library rather than reimplementing them.