Almost anyone can get a RAG demo running: chunk the documents, embed them, retrieve, stuff the prompt. Between that demo and a production system lies an entire engineering chain. These seven checks are the gate every knowledge-base project of ours passes before launch.
1. Was chunking validated on your real documents?
Fixed-size chunking behaves wildly differently on policy documents, tables and FAQs. Validate against your document types before launch, not against a sample PDF.
2. Is retrieval quality measured?
Build an evaluation set of real questions with known source passages, and baseline your retrieval hit rate. Without that number, every later "improvement" is a feeling.
3. Do answers cite their sources?
In an enterprise setting, an answer without a citation is an answer nobody trusts. Citations must link back to the exact passage — that link is the only handle users have for deciding whether to believe the system.
4. Does the system know how to refuse?
When retrieval comes back empty, the model must say "the knowledge base has nothing on this" — not improvise. This single behavior decides whether the system fabricates company policy in front of your executives.
5. Where does permission filtering happen?
Different departments see different documents. Filtering must happen at the retrieval layer — permission-aware queries — not as an afterthought applied to generated text.
6. Is document ingestion automatic?
A knowledge base serving last quarter's policy is worse than no knowledge base. Ingestion should hang off update events at the document source, not depend on someone remembering to re-sync.
7. Is there a feedback loop after launch?
Put helpful / not-helpful buttons in the UI and route the failures back into your evaluation set. RAG quality is operated into existence, not tuned once.
Notice that none of these involve "switching to a stronger model" — production readiness lives mostly in engineering, not in the model. If your knowledge-base project is stuck at the demo stage, reach out for a grounded assessment.