Retrieval-Augmented Generation (RAG)
Make a model answer from your own documents: split them into chunks, embed and index them, retrieve the best chunks for each question, put them in the prompt, and make the answer cite its sources. Plus hybrid search, reranking, and how RAG fails.
Why RAG?
An LLM knows only what was in its training data, up to its cutoff date, and it does not know your clinic's policies, your company's wiki or last week's meeting notes. You could paste every document into every prompt, but that is slow, expensive and limited by the context window.
Retrieval-augmented generation (Lewis et al., 2020) solves this: for each question, retrieve the few most relevant passages from your documents and add them to the prompt, with an instruction to answer only from them and cite them. It is the most common pattern in real LLM apps, because it is cheap, keeps answers grounded and up to date (change a document and the next answer changes), and shows users where answers come from.
The pipeline
Once, ahead of time (indexing):
- Load documents (PDF, web pages, Word files, database rows) and clean the text.
- Chunk: split them into passages. A common starting point is a few hundred tokens per chunk with some overlap between neighbours, splitting on headings and paragraphs rather than mid-sentence. Keep each chunk's source and title.
- Embed every chunk and store the vectors with the text and metadata (embeddings).
For every question (querying):
- Retrieve the top k chunks (say 3 to 8) most similar to the question.
- Augment: build a prompt with the chunks, each labelled with an id, plus the question and the rules ("Answer only from the sources. Cite them like [2]. If they do not contain the answer, say so.").
- Generate the answer, and show the cited sources to the user.
Try it: a tiny RAG system
The knowledge base is eight short passages from a made-up clinic handbook. Ask a question (or pick one), change k, and see which passages are retrieved and the exact prompt that would go to the model. Retrieval here uses keyword scoring (BM25), which runs instantly in the browser.
Interactive: this part runs in your browser.
Making retrieval better
- Hybrid search: combine keyword search (BM25, great for names, codes and rare words like "Medicare") with vector search (great for paraphrases), and merge the two rankings.
- Reranking: retrieve a generous 20 to 50 candidates fast, then reorder them with a slower, more accurate cross-encoder reranker that reads the question and each chunk together, and keep the best few.
- Metadata filters: only search documents the user is allowed to see, the right product, or recent dates. Access control must happen in retrieval, never by asking the model to keep secrets.
- Query rewriting: turn a vague follow-up ("what about for kids?") into a standalone question using the chat history before searching.
- Chunking matters: chunks too small lose context; too big dilute the match and waste tokens. Test a few sizes on your own questions.
How RAG fails, and how to tell which part failed
| Symptom | Likely cause | Fix |
|---|---|---|
| Answer is wrong, and the right passage was not retrieved | retrieval | hybrid search, better chunking, reranking, query rewriting |
| The right passage was retrieved but the answer ignores or contradicts it | generation | clearer instructions, fewer and better chunks, a stronger model |
| Answer includes facts that are in no source | hallucination | require citations, check that each claim is supported, allow "I don't know" |
| Answers are out of date | stale index | re-index changed documents automatically |
Always log the retrieved chunks with each answer: it is the only way to tell retrieval failures from generation failures. The evaluation topic shows how to measure both.
Code: a minimal RAG pipeline
Chunking, retrieval and prompt building run here in the page. The final model call is left as a function you plug your provider into.
import numpy as np, re
handbook = """# Appointments
You can book online, by phone or at reception. Standard appointments are 15 minutes; long appointments are 30 minutes.
# Cancellations
Please cancel at least 24 hours ahead. Late cancellations and missed appointments may be charged a $40 fee.
# Billing
Children under 16 and concession card holders are bulk-billed. Other patients pay at the time of the visit.
# Opening hours
Weekdays 8am to 6pm. Saturdays 9am to 12pm. Closed on Sundays and public holidays."""
# 1. chunk on headings, keeping the title with each chunk
chunks = []
for block in handbook.split("# ")[1:]:
title, body = block.split("\n", 1)
chunks.append({"id": len(chunks) + 1, "title": title.strip(), "text": body.strip()})
# 2. embed (stand-in: word counts; swap in a real embedding model)
tok = lambda t: re.findall(r"[a-z0-9]+", t.lower())
vocab = sorted({w for c in chunks for w in tok(c["title"] + " " + c["text"])})
def embed(t):
v = np.array([tok(t).count(w) for w in vocab], float); n = np.linalg.norm(v)
return v / n if n else v
M = np.stack([embed(c["title"] + " " + c["text"]) for c in chunks])
# 3. retrieve and 4. build the prompt
def build_prompt(question, k=2):
best = np.argsort(-(M @ embed(question)))[:k]
sources = "\n".join(f'<source id="{chunks[i]["id"]}" title="{chunks[i]["title"]}">{chunks[i]["text"]}</source>' for i in best)
return f"""Answer the question using only the sources below.
Cite sources like [1]. If the sources do not contain the answer, say "I don't know".
{sources}
Question: {question}"""
print(build_prompt("Is there a fee if I cancel late?"))
# 5. answer = call_your_model(build_prompt(...)) <- your API call from the previous topicQuiz
1. What is the main purpose of RAG?
2. Users search for a product code like "XR-4471" and vector search misses it. What helps most?
3. The correct passage was retrieved, but the answer still contradicts it. Where is the failure?
4. Some documents are confidential to managers. How should a RAG app protect them?
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- Lewis et al. (2020): Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks ↗ the paper that named RAG
- Ragas documentation ↗ metrics for evaluating RAG pipelines