RAG vs AI Agents: What's the Difference?

"Should we use RAG or an agent?" is one of the most common architecture questions in applied AI, and it is malformed. The two techniques are answers to different questions. RAG answers "the model does not know this — how do I get the knowledge in?" An agent answers "the model knows what to do — how does it actually do it?"

Put concretely: a support assistant that must cite your product documentation is a retrieval problem. A support assistant that must look up an order, check the refund policy, and issue the refund is an agent problem that contains a retrieval problem. The second one needs the first. The first does not need the second.

The knowledge problem and the action problem

Retrieval-augmented generation inserts relevant text into the prompt before the model answers. The control flow is fixed: embed the query, search, assemble a prompt, generate. One model call, one path through the code, every time.

An agent, as described in the complete guide to AI agents, gives the model a set of tools and lets it choose what to call. Control flow is decided at runtime. Retrieval may be one of those tools — but so may writing to a database, calling a payment API or opening a pull request.

The distinction is not sophistication. A well-built RAG pipeline is often the harder engineering job, because chunking, hybrid search and reranking are genuinely subtle while a tool loop is forty lines. The distinction is whether the sequence of steps is fixed.

What a RAG pipeline actually is

Stripped to its core, retrieval is a similarity query plus a prompt assembly step.

from dataclasses import dataclass

@dataclass
class Chunk:
    id: str
    document_title: str
    text: str
    url: str

RETRIEVE_SQL = """
SELECT id, document_title, text, url
FROM doc_chunks
WHERE tenant_id = %(tenant_id)s
ORDER BY embedding <=> %(query_embedding)s::vector
LIMIT %(k)s
"""

def retrieve(cursor, embedder, question: str, tenant_id: str, k: int = 8):
    query_embedding = embedder.embed(question)
    cursor.execute(
        RETRIEVE_SQL,
        {"tenant_id": tenant_id, "query_embedding": query_embedding, "k": k},
    )
    return [Chunk(*row) for row in cursor.fetchall()]

def build_prompt(question: str, chunks: list[Chunk]) -> str:
    context = "\n\n".join(
        f"[{i + 1}] {c.document_title}\n{c.text}" for i, c in enumerate(chunks)
    )
    return (
        "Answer using only the numbered sources below. "
        "Cite sources as [n]. If the sources do not contain the answer, "
        "say so and do not guess.\n\n"
        f"Sources:\n{context}\n\nQuestion: {question}"
    )

Line 17 is the security-critical one and the easiest to omit: the tenant filter lives in the WHERE clause, not in a post-filter and definitely not in the prompt. Vector search will happily return a competitor's document if you let it, and no instruction prevents that.

Line 26 is the honesty clause. Without an explicit permission to fail, a model handed eight irrelevant chunks will synthesise an answer from them. Retrieval quality problems present as confident wrong answers, not as empty responses.

Notice that nothing in this file makes a decision. The number of chunks is a constant, the search runs whether or not it is needed, and the model never gets to say "that wasn't useful, let me search differently."

What an agent adds, and what it costs

Wrap that same retrieval function as a tool and the properties change completely.

SEARCH_DOCS = {
    "name": "search_docs",
    "description": (
        "Search product documentation and return matching passages with source URLs. "
        "Covers setup, billing and API reference. Does NOT cover a specific customer's "
        "account state — use get_account for that. Rephrase and search again if the "
        "first results are off-topic."
    ),
    "input_schema": {
        "type": "object",
        "properties": {
            "query": {
                "type": "string",
                "description": "A focused search phrase, not the user's full question",
            },
            "k": {"type": "integer", "minimum": 1, "maximum": 12, "default": 8},
        },
        "required": ["query"],
    },
}

The model can now search more than once, reformulate after seeing weak results, decide retrieval is unnecessary, or combine documentation with a live account lookup. That is a genuine capability gain.

It is also a genuine cost. The number of model calls is no longer one. Latency becomes variable. The same question asked twice can take different paths. And the guarantee that every answer is grounded in retrieved text disappears — the model may now answer from parametric knowledge without searching at all, which is exactly the behaviour RAG was adopted to prevent.

The comparison, honestly

The "grounding" row is the one that changes architecture decisions. In a fixed pipeline, context is always present because your code put it there. In an agent, grounding is a behaviour you have to verify — and a model that skips the search and answers anyway produces exactly the fluent, unsourced text you were trying to eliminate.

Agentic retrieval: the middle ground

Between the two sits a pattern that is neither: a fixed pipeline with one adaptive step. Retrieval runs unconditionally as it does in RAG, but a small loop is allowed to evaluate the results and search again, up to a low cap.

MAX_SEARCHES = 3

async def answer(question: str, tenant_id: str, model, cursor, embedder) -> dict:
    queries, seen, chunks = [question], set(), []

    for _ in range(MAX_SEARCHES):
        query = queries.pop()
        for chunk in retrieve(cursor, embedder, query, tenant_id):
            if chunk.id not in seen:
                seen.add(chunk.id)
                chunks.append(chunk)

        verdict = await model.classify(
            instruction=(
                "Do these passages contain enough information to answer the question? "
                "Reply 'sufficient', or 'insufficient: <better search phrase>'."
            ),
            question=question,
            passages=[c.text for c in chunks],
        )
        if verdict.startswith("sufficient"):
            break
        queries.append(verdict.split(":", 1)[1].strip())

    return {
        "answer": await model.generate(build_prompt(question, chunks)),
        "sources": [{"title": c.document_title, "url": c.url} for c in chunks],
    }

This keeps the property that made RAG attractive — retrieval always happens, so the answer is always grounded — while recovering from the single biggest RAG failure, which is a user question phrased nothing like the documentation. The loop is bounded at three searches, so the cost ceiling stays knowable.

Most systems described as "agentic RAG" are this, not a full tool-calling agent. That is usually the right call.

Their failure modes barely overlap

RAG fails at the retrieval boundary. The chunk containing the answer was split down the middle, so neither half scores well. The user asked about "invoicing" and the docs say "billing", and pure vector similarity does not bridge it as reliably as people assume — which is why hybrid search combining lexical and vector scoring is standard practice. Or the top result is a deprecated page, because relevance ranking has no concept of freshness unless you gave it one.

Agents fail at the decision boundary. The model called get_account when the answer was in the docs. It searched once, got mediocre results, and answered anyway. It called a write tool it should not have had access to. These are not retrieval problems and no amount of chunking work fixes them.

The practical consequence: you cannot reuse a RAG evaluation suite on an agent. RAG evaluation asks whether the right passages were retrieved and whether the answer is faithful to them. Agent evaluation asks whether the right tools were called in a sensible order and whether the world ended up in the correct state.

Choosing

Use a fixed RAG pipeline when the job is answering questions over a corpus, every answer must be attributable, latency matters, and nothing needs to be written. Documentation assistants, internal search, policy lookup.

Add an adaptive retrieval loop when the corpus is broad enough that a single query often misses, but the job is still purely to answer.

Use an agent when answering requires touching more than one system, the required steps vary per request, or the outcome is an action rather than a paragraph. Refunds, triage, provisioning, investigation.

Do not use an agent when the sequence is known. Deciding that boundary is the whole of AI agents versus traditional automation, and the honest conclusion there is that most "agents" are pipelines wearing a costume.

Building either one properly

Whichever you pick, the evaluation harness is the part that determines whether the system improves or merely changes. For retrieval, that means a fixed question set with known-correct source documents, scored on whether those documents appear in the retrieved set at all — a generation problem downstream of a retrieval miss is unfixable by prompting. For agents, it means recorded scenarios scored on tool trajectory and final system state.

Both also need the same production scaffolding: tenant isolation enforced in the query layer, transcripts persisted for debugging, cost attributed per request, and a deliberate answer for what happens when the model returns nothing useful. That scaffolding is the subject of building production-ready AI applications, and it is where more projects fail than at the choice of architecture. If you want to see the agent side end to end, building an AI agent with Next.js walks through a working loop.