"It seems to work" is not a measurement. RAG systems degrade quietly, so going to production without measurement carries risk.
01. Two Separate Measurements
Retrieval quality and answer quality must be measured separately. If the right document isn't found, the problem is retrieval; if it is found but the answer is wrong, the problem is the prompt or the model.
02. Building a Question Set
Collect fifty to a hundred real user questions together with their expected answers. That set becomes your system's exam paper, rerun after every change.
03. Groundedness
Is every claim in the answer present in the retrieved documents? This check is the most practical way to catch a fabricated answer. A model speaking from its own general knowledge is unacceptable in an enterprise setting.
04. Knowing What's Out of Scope
Can the system say it doesn't know? A system that produces an answer to every question loses user trust at the first wrong one.
05. Regression Testing
When documents are added, the model changes or the prompt is updated, the same set must be rerun. Some changes believed to be improvements break something elsewhere.
06. Monitoring in Production
Collect the answers users flag negatively and review them regularly. The most valuable test data comes from there.