Retrieval-Augmented Generation (RAG)

A RAG pipeline has two halves that fail for different reasons. The indexing half loads source documents, splits them into chunks, embeds each chunk and writes it to a store — decisions about chunk size, overlap and what metadata travels with each chunk are made here and are expensive to change later. The query half embeds the incoming question, retrieves the nearest chunks, optionally reranks them, assembles a prompt and generates the answer. When RAG disappoints, the cause is usually retrieval, not generation: the right passage never made it into the context, so no amount of prompt tuning will recover it.

RAG matters whenever the answer depends on information the model was not trained on or that changes faster than any training cycle: internal documentation, product catalogues, ticket histories, contracts, last week’s changelog. It also matters for attribution, because a retrieved chunk carries a source you can link to, which turns an unverifiable assertion into a checkable one.

Two misconceptions are worth correcting. The first is that a long context window makes RAG obsolete — a large window changes what you can afford to include, but it does not tell you which of ten million documents is relevant, and paying to process irrelevant text is both slower and less accurate. The second is that RAG eliminates hallucination. It reduces one cause and introduces another: given a confident but irrelevant retrieval, a model will happily build a fluent answer on top of the wrong passage.