RAG explained: how to make AI answer from your own documents

Ask a general AI model about your company’s refund policy and it will give you an answer. It will sound confident, and it may be completely made up. The model was trained on the public internet, not on your policy documents. When it does not know something, it produces the most plausible-sounding text instead. This is what people mean by "hallucination".

Retrieval-augmented generation (RAG) fixes this by looking things up first. Before the model answers, the system finds the relevant passages in your own documents and hands them over with the question, along with an instruction to answer only from them.

#Why not just train the model on our documents?

Fine-tuning a model teaches it style and patterns well, but it is a poor way to store facts. It is expensive, it goes out of date the moment your price list changes, and you cannot easily see which document an answer came from. RAG keeps your knowledge in documents you can edit, and the model reads the current version every time.

#How RAG works, step by step

1. 01

   Collect

   Gather the sources: PDFs, Word files, help-centre articles, website pages, spreadsheets, internal wikis.

2. 02

   Chunk

   Split each document into passages of a few paragraphs, following its headings, and keep the title and source with every chunk.

3. 03

   Embed

   Convert each chunk into a vector, a list of numbers that captures its meaning. Passages with similar meaning end up close together.

4. 04

   Store

   Save the vectors with their metadata (source, date, department, who may see it) in a vector database, or in Postgres with pgvector.

5. 05

   Retrieve

   When a question arrives, embed it the same way and find the closest chunks. Combine with keyword search so exact codes and names are not missed.

6. 06

   Answer

   Give the model the question, the retrieved chunks and clear rules: answer only from these passages, cite them, and say so if they do not cover the question.

Answer the customer's question using ONLY the passages below.
Cite the passage number after each fact, like [2].
If the passages do not contain the answer, say you don't know
and offer to connect the customer with a person.
Reply in the language the customer used.

Passages:
[1] Delivery inside Dhaka: 1–2 working days ...
[2] Delivery outside Dhaka: 3–5 working days ...

Question: ঢাকার বাইরে ডেলিভারিতে কতদিন লাগে?

#Where RAG goes wrong

Bad sources

If there are two versions of the price list in the system, the AI will sometimes quote the old one. No retrieval technique fixes contradictory documents. The fix is organisational: one owner per document, one source of truth, and old versions removed rather than left alongside the new ones.

Bad chunks

Split a table down the middle, or separate a paragraph from the heading that gives it meaning, and the retrieved passage stops making sense. Chunking that follows the document’s structure, and adds the section title to each chunk, makes a large difference.

Bad retrieval

The customer asks how to "cancel"; the policy talks about "termination". Pure keyword search misses it; pure vector search can miss exact product codes. Hybrid search (both together), re-ranking the top results, and rewriting vague questions before searching all help.

No access control

If HR documents are in the same index as product manuals, a customer might get an answer drawn from salary policy. Permissions must be applied at retrieval time, by filtering on who is asking, not by hoping the model stays quiet.

#How to tell if your RAG is any good

Build a test set of real questions with known correct answers and their source documents, then check three things for each one:

• Retrieval: did the right passage come back in the top results?
• Faithfulness: does the answer say only what the passage says?
• Honesty: when the documents do not cover a question, does the system say so instead of guessing?

Run the set before and after every change to documents, chunking or prompts. It is the only reliable way to know whether a change helped. For the full method, read RAG evaluation.

#Mixed languages

For many teams in Bangladesh, documents are written in English but questions arrive in Bangla or Banglish. Multilingual embedding models can match a Bangla question to an English passage, but this needs testing with real questions, not assumptions. We include mixed-language questions in every test set.

#Where to start

Start with one department and one well-kept set of documents, for example HR policies for staff, or a product catalogue for customers. Clean those documents first, launch with sources shown on every answer, and expand once people trust it. Our AI knowledge base & RAG page covers what a first project looks like.