Retrieval-augmented generation
Retrieval-augmented generation finds the passages of your own documents that bear on a question and puts them in the model's prompt, so the answer can draw on text you control and can cite.
What it is
Retrieval-augmented generation (RAG) answers a question in two moves: a retriever finds the parts of a knowledge base that bear on the question, and a language model writes the answer with those parts placed in its prompt. The name comes from a 2020 paper by Lewis and colleagues, who paired a sequence-to-sequence model, whose knowledge sits in its weights, with a dense vector index of Wikipedia read by a neural retriever. Their motivation still explains why teams reach for the pattern: knowledge baked into weights is hard to update and hard to attribute, while retrieved text can be refreshed and cited.
In practice there is an ingestion half that runs once per document and a query half that runs per question. Ingestion: clean the documents, split them into chunks (usually a few hundred tokens each), run each chunk through an embedding model to get a vector that encodes its meaning, and store vector, text and metadata in a vector index. Query: embed the question the same way, fetch the chunks whose vectors sit closest to it, add them to the prompt as context beside the question, and send that to the model. To the user it looks like any chat with a model; what changed is what the model was shown before it answered.
| Component | Job | Decision you own |
|---|---|---|
Connectors and parsing | Pull PDFs, wikis, tickets and database rows into text | Which sources, and how often they re-sync |
Chunker | Split documents into retrievable units | Size, overlap, and whether to prefix document context |
Embedding model | Turn chunks and questions into vectors | Which model; re-embed the whole corpus if you change it |
Vector index | Store vectors with text and metadata; nearest-neighbor search | Exact or approximate search; which metadata to filter on |
Retriever and reranker | Fetch candidates, merge rankings, score the best | Hybrid or just vectors; how many candidates to rerank |
Orchestrator | Run the query flow and build the prompt | How many chunks to include; how to ask for citations |
Guardrails and access control | Keep out unauthorized content and unsafe output | Where the permission filter runs |
Chunking exists for two reasons. Embedding and chat models cap their input: the text-embedding-3-small model, for example, accepts at most 8,191 tokens. And a page that covers many topics embeds poorly as one vector, so a finer split retrieves better even when the page would fit. Azure AI Search's guidance suggests starting at 512 tokens per chunk with a 25% overlap (128 tokens), and splitting on structure such as headings or sentence ends when the content has it. A chunk cut from the middle of a report loses what it was about: a line saying revenue grew 3% carries no company and no quarter. Anthropic's contextual retrieval addresses this by having a model write a 50 to 100 token sentence that situates each chunk in its document, prepended before embedding and indexing. In their tests that cut the share of questions whose relevant chunk was missing from the top 20 results from 5.7% to 3.7% with embeddings on their own.
Embedding search matches meaning, which is why it can find a paragraph about time off when the user typed PTO, and also why it can miss an exact token such as a part number or an error code. BM25, a keyword ranking function built on term frequency with a correction for document length, finds the exact string. Hybrid retrieval runs both and merges the two ranked lists with rank fusion; a search service can run the keyword and vector queries in parallel from one request and return a single merged result set. A reranker then takes a wide candidate set and scores each chunk against the question with a dedicated reranking model. Anthropic's experiment fetched 150 candidates and kept 20, and reranking on top of contextual embeddings plus BM25 brought the top-20 miss rate down to 1.9%. The reranker is an extra model call per question, so the candidate count is a latency knob you should name.
The vector index trades recall for speed. pgvector 0.8.7 searches exactly by default, which scans the table and misses nothing; adding an HNSW or IVFFlat index switches to approximate nearest-neighbor search and can change which rows a query returns. HNSW, a layered graph, answers faster at a given recall but builds more slowly and uses more memory. IVFFlat groups vectors into lists and searches the nearest few; it builds faster and smaller, ranks lower on the speed-recall curve, and its documentation advises building it after the table holds data so the lists reflect real vectors. Filters interact badly with approximate indexes: pgvector applies a WHERE clause after the index scan, so with the default candidate list of 40 and a filter matching 10% of rows you get about 4 results, unless you enable iterative scans or build a partial index for that filter value.
When to use it
Reach for RAG when the knowledge the model needs is larger than its context window, when it changes faster than you could retrain a model (an indexer that re-syncs changed documents keeps the index fresh), when the answer must point at its source, or when different users are allowed to see different documents. Query understanding, token limits, response time and access control are the usual difficulties, and they make a good checklist for the parts of the design to talk through.
Two of those shape the design the most. Freshness: the index lags the source, because a changed document stays invisible to retrieval until the next ingestion run has re-chunked and re-embedded it, so describe how that run is triggered and how long the lag is. Permissions: a chunk inherits the access rules of its document, and the usual approach is document-level security trimming, a filter applied inside the search query so unauthorized chunks are dropped before any text reaches the prompt. A filter applied after generation is too late, because the model has already read the text.
When not to
When the whole knowledge base is small, put it in the prompt. Anthropic's guidance is that a corpus under about 200,000 tokens, roughly 500 pages, can go into the prompt as is, and prompt caching makes re-sending it cheaper and faster. You skip chunking, embeddings, an index and a whole class of retrieval misses. Say this out loud when the problem statement is a product manual or a single policy handbook.
When the question is a lookup, query the system of record. If a user asks for the status of their order, the orders service already answers that exactly, under whatever access rules it applies; a similarity search over documents adds a fuzzy step where none is wanted. And when the model's job is tone, format or a skill rather than facts (rewrite this ticket reply in our house style), a few examples in the prompt are the simpler first thing to try.
When the latency budget is tight, keep the pipeline short. Users expect answers within a few seconds, and each stage you add (a model that rewrites the question into subqueries, a reranker over hundreds of candidates) spends part of that budget. Azure AI Search's own guidance recommends its classic single-query pattern when simplicity and speed outrank the last bit of relevance, and scoring more candidates in a reranker adds latency and cost.
Compared
| Approach | What it gets right | Where it falls short | Pick it when |
|---|---|---|---|
Whole corpus in the prompt | No retrieval misses; nothing to index or keep in sync | Capped by the context window; each question pays for the full corpus | The corpus is under about 200K tokens and fairly stable |
Embedding search on its own | Matches meaning across different wording | Misses exact identifiers; chunks lose document context | Natural-language questions over prose where identifiers are rare |
Keyword (BM25) on its own | Exact hits on codes, names and terms; no embedding model to run | Fails when the question's words differ from the document's | Logs, part numbers, legal citations, or a first version on an existing search engine |
Hybrid plus reranker | Catches both paraphrases and exact matches; the reranker trims the noise | Two retrieval paths to tune and an extra model call per question | A large mixed corpus where a missed chunk is costly |
Agentic retrieval (a model plans subqueries) | Handles vague, multi-part, conversational questions | Adds planning calls, so usually slower and dearer; more places to fail | Complex questions where accuracy outranks response time |
In a picture
How to say it in an interview
The common mistake
Quiz
Related
Sources
- arXiv: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020)
- AWS Prescriptive Guidance: Understanding Retrieval Augmented Generation
- Microsoft Learn: Retrieval-augmented generation (RAG) in Azure AI Search
- Microsoft Learn: Chunk large documents for RAG and vector search in Azure AI Search
- Anthropic Engineering: Introducing Contextual Retrieval
- pgvector README (v0.8.7): Open-source vector similarity search for Postgres