Agentic RAG: When Your Retriever Should Think Before It Searches

A vanilla RAG pipeline treats every question the same way: embed the raw user query, retrieve top-K, feed to the LLM. That works for 'What is our refund policy?'. It falls flat on:

• Multi-hop questions ('Which of our top-10 customers has an expiring contract in Q4?')
• Ambiguous queries ('Show me the report' — which one?)
• Questions where the phrasing is nothing like the source ('How do I cancel?' vs a doc titled 'Subscription lifecycle management')
• Questions that need a different data source (SQL, live API, web)

Agentic RAG adds a reasoning step where the LLM decides how to retrieve, not just what to say afterward.

The single highest-ROI upgrade. Before retrieval, ask a small LLM to rewrite the query into 1-3 search-optimized variants:

REWRITE_PROMPT = '''
Rewrite the user question into 1-3 concise search queries that would
retrieve relevant documents. Include synonyms and expanded acronyms.
Return JSON: {"queries": ["..."]}

Question: {q}
'''

queries = json.loads(llm(REWRITE_PROMPT.format(q=user_q)))['queries']
hits = fuse([retriever.search(q, k=10) for q in queries])

Cheap (a small model, ~200 tokens), fast (<500ms), and reliably lifts Recall@10 by 10-20 points on natural user queries.

For multi-hop questions, decompose into simpler sub-questions, answer each independently, then compose the final answer.

Example: 'Which customer had the most support tickets last month, and what are their top complaint categories?' becomes:

1. Sub-Q1: 'Which customer had the most support tickets last month?' → SQL retriever → 'ACME Corp, 47 tickets'
2. Sub-Q2: 'What are ACME Corp's top complaint categories?' → vector retriever over tickets → top 3 tags
3. Compose: LLM combines both answers.

Frameworks like LangGraph, LlamaIndex QueryEngine, and DSPy have first-class support for this. The pattern is called sub-question query engine in LlamaIndex.

Real questions need different data sources. A router agent picks between them:

ROUTE = '''
Given the question, choose ONE retriever:
- 'docs': product documentation, policies, how-tos
- 'sql': live customer data, orders, invoices
- 'web': news, current events, third-party info
Return JSON: {"retriever": "docs|sql|web", "reason": "..."}

Question: {q}
'''

Keep the router prompt tight and the retriever set small (3-5). Every added option makes routing accuracy drop. When the router picks wrong, your best mitigation is the next pattern.

After retrieval, have a small LLM grade whether the passages actually answer the question. If not, rewrite or re-route and retry.

loop up to 3 times:
  passages = retriever.search(query)
  grade = grader_llm(question, passages)
  if grade.relevant: break
  query = rewriter_llm(question, previous_failed_query=query, passages=passages)
answer = answerer_llm(question, passages)

The loop turns a 60% correct system into an 85% correct one on hard queries. It also increases cost by ~2x on average (many queries pass on first try) and latency by 1-3 seconds when it retries. Use it on high-stakes flows, skip it on low-margin ones.

A rough per-query comparison for a well-optimized system:

• Static RAG: 1 retrieval + 1 LLM call. Latency ~1-2s, cost ~$0.001.
• + query rewriting: adds ~500ms and ~$0.0002. Almost always worth it.
• + decomposition (avg 2 sub-questions): ~3-5s, ~$0.003. Worth it for multi-hop-heavy corpora.
• + self-correction (worst case): ~6-10s, ~$0.008. Worth it only when accuracy is the constraint.

Measure on your own eval set. The right amount of 'agentic' is the amount that improves the metric you actually care about, at a cost you can afford.