RAG evaluation: how to test a chatbot that answers from your documents

A RAG chatbot answers questions from your own documents. Before the model writes anything, the system searches your files, picks the most relevant passages and hands them to the model with an instruction to answer only from them. Our RAG explainer covers how that works. This article covers the next question: how do you know it works well enough to put in front of staff or customers?

RAG evaluation is the process of checking, on a fixed set of questions, whether the system finds the right passages and then answers faithfully from them. Done well, it tells you not only that something is wrong, but where.

#Test the two halves separately

Every RAG answer goes through two steps, and either can fail:

• Retrieval: the search step. Did the right passage come back near the top of the results?
• Generation: the writing step. Given the passages, did the model answer correctly, using only what they say?

If you only grade the final answer, a wrong answer could mean either. The fixes are completely different. Bad retrieval is fixed with better chunking, better search or cleaner documents. Bad generation is fixed with better instructions, a stronger model or stricter citation rules. Measuring both halves on every question stops you from tuning the prompt when the real problem is search.

#Step 1: Build the test set

A RAG test set is a list of questions, each with the correct answer and the passage (or passages) that contain it. We aim for real questions first: search logs, support tickets, the questions new staff ask in their first month. Then we add the cases that usually break things:

• Paraphrases. The same question in words that do not appear in the document ("send it back" instead of "return").
• Two-document questions. The answer needs a policy from one file and a price from another.
• Not covered. Questions the documents do not answer. The right response is "I don’t know", not a guess.
• Old versions. Topics where an outdated document is still in the library. The system must prefer the current one.
• Restricted content. Questions a given user should not be able to get answers to, such as salary data for a non-HR user.
• Mixed language. Questions in Bangla or Banglish about documents written in English, or the other way round, if your users write that way.

- id: sale-item-return
  question: "Can I return something I bought in the sale?"
  user_role: customer_support
  expected_sources: ["returns-policy.pdf#p3"]
  must_not_use: ["returns-policy-2024.pdf"]   # superseded
  answer_must_include: ["7 days", "unused", "receipt"]

- id: not-covered-warranty-abroad
  question: "Is the warranty valid if I move abroad?"
  user_role: customer_support
  expected_sources: []          # documents do not cover this
  expected_behaviour: says_unknown_and_offers_handover

#Step 2: Measure retrieval

Retrieval can be graded automatically, because each question already lists the passages it needs. The useful numbers are:

Set k to the number of passages the model actually receives. A passage at rank 12 does not help if only the top 5 are passed on.

#Step 3: Measure generation

Now grade the answers, given what was retrieved. These checks need judgement, so we use a second model as a grader with a written rubric and the source passages in front of it, and a person reviews a sample of its verdicts.

• Faithfulness. Does every claim in the answer appear in the retrieved passages? An answer can be true in the real world and still be unfaithful, if it came from the model’s general knowledge instead of your documents.
• Answer correctness. Does the answer match the one the document owner confirmed, including the details that matter (amounts, deadlines, conditions)?
• Citation accuracy. Do the sources it shows actually support what it says? A citation to the wrong page is worse than none, because it looks trustworthy.
• Honest abstention. For not-covered questions, does it say it does not know, and offer a next step? Also track the opposite: how often it says "I don’t know" when the answer was right there.

#Reading the results: symptom, cause, fix

#Step 4: Re-run on every change

RAG systems change more often than people expect. New documents are added, chunk sizes are tuned, embedding models are upgraded, prompts are edited, and the model provider ships new versions. Any one of these can improve some answers and quietly break others. The test set runs on every change, and each run is compared with the last one, not just with a threshold.

After launch, add real questions that went wrong to the set. Flag conversations where users rephrase the same question, rate an answer badly or ask for a person: those are usually retrieval failures. How we turn these numbers into a go-live decision is covered in AI agent evaluation metrics.

#A starter plan

1. 01

   Collect 50 real questions

   From tickets, emails or staff, with the passage that answers each one confirmed by the document owner.

2. 02

   Add the hard cases

   Paraphrases, not-covered questions, old versions and at least one restricted question per user role.

3. 03

   Grade both halves

   Hit rate and permission leaks for retrieval; faithfulness, correctness and abstention for answers.

4. 04

   Fix the biggest gap first

   Usually retrieval. There is little point polishing answers the model never had the passage for.

This is how we test every AI knowledge base and document-answering chatbot we build. For the wider testing process around agents that also take actions, read how to test an AI agent before go-live.