Skip to content

Retrieval-augmented generation

Retrieval-augmented generation finds the passages of your own documents that bear on a question and puts them in the model's prompt, so the answer can draw on text you control and can cite.

On this page

What it is

Retrieval-augmented generation (RAG) answers a question in two moves: a retriever finds the parts of a knowledge base that bear on the question, and a language model writes the answer with those parts placed in its prompt. The name comes from a 2020 paper by Lewis and colleagues, who paired a sequence-to-sequence model, whose knowledge sits in its weights, with a dense vector index of Wikipedia read by a neural retriever. Their motivation still explains why teams reach for the pattern: knowledge baked into weights is hard to update and hard to attribute, while retrieved text can be refreshed and cited.

In practice there is an ingestion half that runs once per document and a query half that runs per question. Ingestion: clean the documents, split them into chunks (usually a few hundred tokens each), run each chunk through an embedding model to get a vector that encodes its meaning, and store vector, text and metadata in a vector index. Query: embed the question the same way, fetch the chunks whose vectors sit closest to it, add them to the prompt as context beside the question, and send that to the model. To the user it looks like any chat with a model; what changed is what the model was shown before it answered.

ComponentJobDecision you own

Connectors and parsing

Pull PDFs, wikis, tickets and database rows into text

Which sources, and how often they re-sync

Chunker

Split documents into retrievable units

Size, overlap, and whether to prefix document context

Embedding model

Turn chunks and questions into vectors

Which model; re-embed the whole corpus if you change it

Vector index

Store vectors with text and metadata; nearest-neighbor search

Exact or approximate search; which metadata to filter on

Retriever and reranker

Fetch candidates, merge rankings, score the best

Hybrid or just vectors; how many candidates to rerank

Orchestrator

Run the query flow and build the prompt

How many chunks to include; how to ask for citations

Guardrails and access control

Keep out unauthorized content and unsafe output

Where the permission filter runs

Chunking exists for two reasons. Embedding and chat models cap their input: the text-embedding-3-small model, for example, accepts at most 8,191 tokens. And a page that covers many topics embeds poorly as one vector, so a finer split retrieves better even when the page would fit. Azure AI Search's guidance suggests starting at 512 tokens per chunk with a 25% overlap (128 tokens), and splitting on structure such as headings or sentence ends when the content has it. A chunk cut from the middle of a report loses what it was about: a line saying revenue grew 3% carries no company and no quarter. Anthropic's contextual retrieval addresses this by having a model write a 50 to 100 token sentence that situates each chunk in its document, prepended before embedding and indexing. In their tests that cut the share of questions whose relevant chunk was missing from the top 20 results from 5.7% to 3.7% with embeddings on their own.

Embedding search matches meaning, which is why it can find a paragraph about time off when the user typed PTO, and also why it can miss an exact token such as a part number or an error code. BM25, a keyword ranking function built on term frequency with a correction for document length, finds the exact string. Hybrid retrieval runs both and merges the two ranked lists with rank fusion; a search service can run the keyword and vector queries in parallel from one request and return a single merged result set. A reranker then takes a wide candidate set and scores each chunk against the question with a dedicated reranking model. Anthropic's experiment fetched 150 candidates and kept 20, and reranking on top of contextual embeddings plus BM25 brought the top-20 miss rate down to 1.9%. The reranker is an extra model call per question, so the candidate count is a latency knob you should name.

The vector index trades recall for speed. pgvector 0.8.7 searches exactly by default, which scans the table and misses nothing; adding an HNSW or IVFFlat index switches to approximate nearest-neighbor search and can change which rows a query returns. HNSW, a layered graph, answers faster at a given recall but builds more slowly and uses more memory. IVFFlat groups vectors into lists and searches the nearest few; it builds faster and smaller, ranks lower on the speed-recall curve, and its documentation advises building it after the table holds data so the lists reflect real vectors. Filters interact badly with approximate indexes: pgvector applies a WHERE clause after the index scan, so with the default candidate list of 40 and a filter matching 10% of rows you get about 4 results, unless you enable iterative scans or build a partial index for that filter value.

When to use it

Reach for RAG when the knowledge the model needs is larger than its context window, when it changes faster than you could retrain a model (an indexer that re-syncs changed documents keeps the index fresh), when the answer must point at its source, or when different users are allowed to see different documents. Query understanding, token limits, response time and access control are the usual difficulties, and they make a good checklist for the parts of the design to talk through.

Two of those shape the design the most. Freshness: the index lags the source, because a changed document stays invisible to retrieval until the next ingestion run has re-chunked and re-embedded it, so describe how that run is triggered and how long the lag is. Permissions: a chunk inherits the access rules of its document, and the usual approach is document-level security trimming, a filter applied inside the search query so unauthorized chunks are dropped before any text reaches the prompt. A filter applied after generation is too late, because the model has already read the text.

When not to

When the whole knowledge base is small, put it in the prompt. Anthropic's guidance is that a corpus under about 200,000 tokens, roughly 500 pages, can go into the prompt as is, and prompt caching makes re-sending it cheaper and faster. You skip chunking, embeddings, an index and a whole class of retrieval misses. Say this out loud when the problem statement is a product manual or a single policy handbook.

When the question is a lookup, query the system of record. If a user asks for the status of their order, the orders service already answers that exactly, under whatever access rules it applies; a similarity search over documents adds a fuzzy step where none is wanted. And when the model's job is tone, format or a skill rather than facts (rewrite this ticket reply in our house style), a few examples in the prompt are the simpler first thing to try.

When the latency budget is tight, keep the pipeline short. Users expect answers within a few seconds, and each stage you add (a model that rewrites the question into subqueries, a reranker over hundreds of candidates) spends part of that budget. Azure AI Search's own guidance recommends its classic single-query pattern when simplicity and speed outrank the last bit of relevance, and scoring more candidates in a reranker adds latency and cost.

Compared

ApproachWhat it gets rightWhere it falls shortPick it when

Whole corpus in the prompt

No retrieval misses; nothing to index or keep in sync

Capped by the context window; each question pays for the full corpus

The corpus is under about 200K tokens and fairly stable

Embedding search on its own

Matches meaning across different wording

Misses exact identifiers; chunks lose document context

Natural-language questions over prose where identifiers are rare

Keyword (BM25) on its own

Exact hits on codes, names and terms; no embedding model to run

Fails when the question's words differ from the document's

Logs, part numbers, legal citations, or a first version on an existing search engine

Hybrid plus reranker

Catches both paraphrases and exact matches; the reranker trims the noise

Two retrieval paths to tune and an extra model call per question

A large mixed corpus where a missed chunk is costly

Agentic retrieval (a model plans subqueries)

Handles vague, multi-part, conversational questions

Adds planning calls, so usually slower and dearer; more places to fail

Complex questions where accuracy outranks response time

In a picture

embedkeywordstop 150User questionVector indexBM25 indexFuse rankingsRerank to top 20Question + chunksLLMAnswer withsources
The query half of a hybrid pipeline: both indexes return candidates, rank fusion merges them, a reranker keeps the best few, and the model answers from those chunks.

How to say it in an interview

The common mistake

Quiz

Users search a support knowledge base for error codes like TS-999 and embedding search keeps returning general articles about errors. What is the most direct fix?
A team wants a chatbot over one 300-page product manual that changes twice a year. What is the simplest design worth proposing first?
A chunk reading 'revenue grew 3% over the previous quarter' is retrieved for questions about the wrong company. Which change attacks the cause?
A filtered query over an HNSW index in pgvector asks for 10 rows where tenant_id = 7, but returns 4. Why?

Related

Sources

  1. arXiv: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020)
  2. AWS Prescriptive Guidance: Understanding Retrieval Augmented Generation
  3. Microsoft Learn: Retrieval-augmented generation (RAG) in Azure AI Search
  4. Microsoft Learn: Chunk large documents for RAG and vector search in Azure AI Search
  5. Anthropic Engineering: Introducing Contextual Retrieval
  6. pgvector README (v0.8.7): Open-source vector similarity search for Postgres
Sign in to report a mistake