RAG That Actually Works: Chunking, Hybrid Search, Reranking, and Evals

The canonical RAG demo — embed the docs, cosine search, stuff the top-k, ask GPT — works beautifully on the FAQ page and collapses the moment users ask anything that requires reasoning, comparison, or numbers. If you've watched an internal chatbot go from “wow” to “can we turn it off?” in a month, this article is the difference between those two states.

Naïve RAG fails for five predictable reasons: bad chunk boundaries, embedding-only retrieval, no reranker, no citations, and no evals. Fix those five and you have a system that survives real traffic; fix only one or two and you have a demo with better slides.

The rest of this article walks the full production pipeline end-to-end: ingestion, chunking, embeddings, indexing, hybrid retrieval, reranking, query rewriting, grounded generation, citations, evaluation, observability, and cost engineering. Each section links out to the deeper articles on the topic when there's one worth reading.

Retrieval quality is upper-bounded by ingestion quality. If your PDFs come in as scrambled text with headers cut in half, no embedding model in the world will save you. Invest here first.

• HTML / Markdown — parse with a real DOM/AST parser, strip nav and footers, and preserve heading hierarchy as metadata. Never regex the HTML.
• PDFs — use a layout-aware extractor (Unstructured, LayoutLMv3, Azure Document Intelligence, or pdfplumber for simple two-column layouts). Raw pdftotext loses tables and footnotes.
• Code — parse with tree-sitter so you can split on function/class boundaries. See Designing REST APIs LLMs can use for why structured metadata matters when the retriever is a model.
• Slack / tickets / chat — group by thread, deduplicate the reply-chain quoting, and attach the original question as context to every reply chunk.
• Databases — export views as row-level markdown snippets; do not embed raw SQL rows.

Store the raw source alongside the extracted text. When you later change chunker or embedder, you must be able to re-derive chunks without re-fetching the world. The modern data stack pieces on this — treat your document lake exactly like an event lake: immutable raw + versioned transforms.

Fixed-size 512-token chunks with 50-token overlap is the wrong default for anything more structured than a novel. It splits ideas mid-sentence and forces the retriever to guess which fragment contains the answer.

Three rules govern good chunking:

1. Split on structure, not tokens. H2/H3, function definitions, table rows, list items, ticket messages — these are natural units of meaning.
2. Keep the parent context. A chunk that says “The rate limit is 60 requests per minute” is useless without knowing which API. Prepend the section path (API › Billing › Rate limits) into the chunk's metadata and into the text the embedder sees.
3. Generate a one-sentence context header. Use a cheap model to write one sentence describing what the chunk covers, then prepend it. This one trick lifts retrieval recall more than upgrading from text-embedding-3-small to text-embedding-3-large in every domain I've measured.

def contextualize(chunk: str, section_path: str, doc_title: str) -> str:
    # 1 call per chunk, batched, using a small model (gpt-4o-mini / haiku).
    prompt = (
        f"Doc: {doc_title}\nSection: {section_path}\n\nChunk:\n{chunk}\n\n"
        "Write ONE sentence describing what this chunk explains. No preamble."
    )
    header = cheap_llm(prompt)
    return f"[{doc_title} › {section_path}] {header}\n\n{chunk}"

Target chunk size after contextualization: 200–500 tokens for prose, 100–300 for code, and one row per chunk for tables. Overlap only when a natural boundary is missing (e.g. long uninterrupted paragraphs).

The MTEB leaderboard is a lie about your domain. What matters is recall@10 on your queries against your corpus. Pick two candidate models, embed a 5,000-chunk sample with each, run 50 real user queries, and score.

For multilingual work (Arabic, German, mixed English/Arabic), see Building multilingual AI for Arabic and English — most English-only embedders lose 15–25 points of recall on Arabic queries, and the fix is not just “add a translation layer.”

Embed dimension matters less than you think below 1024 dims. Above that, index cost and memory footprint climb faster than accuracy — see the pgvector benchmarks in the vector database piece linked above.

Below ~500k chunks, Postgres + pgvector with an HNSW index is almost always the right answer. You get transactional writes, joins, RLS, familiar backups, and a query planner your team already understands. Read Postgres for product engineers if you're not already comfortable pushing Postgres this hard.

Above that scale, or when you need distributed sharding or advanced hybrid features, evaluate a dedicated store (Qdrant, Weaviate, Vespa, Pinecone). Migrating from pgvector when you actually hit the wall is a 1-week job. Starting on Pinecone before you need it costs you 12 months of iteration speed.

-- pgvector HNSW index, tuned for 1M rows and p95 < 50ms
create index chunks_embedding_hnsw
  on chunks using hnsw (embedding vector_cosine_ops)
  with (m = 16, ef_construction = 64);

-- At query time
set hnsw.ef_search = 40; -- higher = better recall, slower

-- Always fetch metadata in the same query; avoid a second round-trip
select id, doc_id, section_path, text,
       1 - (embedding <=> $1) as score
from chunks
where tenant_id = $2                       -- always filter tenant first
order by embedding <=> $1
limit 40;

Dense retrieval is great at meaning and terrible at strings that don't appear in the training data — product SKUs, error codes, acronyms, version numbers. BM25 is the opposite. Running both in parallel and fusing the ranks with Reciprocal Rank Fusion (RRF) beats either alone in every benchmark I've run.

with vec as (
  select id, row_number() over (order by embedding <=> $1) as rnk
  from chunks where tenant_id = $3 order by embedding <=> $1 limit 40
),
bm as (
  select id, row_number() over (order by ts_rank(fts, plainto_tsquery($2)) desc) as rnk
  from chunks
  where tenant_id = $3 and fts @@ plainto_tsquery($2)
  limit 40
)
select coalesce(vec.id, bm.id) as id,
       coalesce(1.0/(60 + vec.rnk), 0) + coalesce(1.0/(60 + bm.rnk), 0) as rrf
from vec full outer join bm using (id)
order by rrf desc
limit 20;

The constant 60 in RRF is a well-known tuning knob; anywhere from 10 to 100 works. Higher values give BM25 more weight on tail results, which usually helps for technical corpora full of identifiers.

If you're running outside Postgres, most vector stores now ship hybrid natively (Qdrant, Weaviate, Vespa, Elasticsearch dense_vector). Use the built-in — it's faster than doing RRF at the application layer.

Retrieval gets you a shortlist of 20–40 candidates. A cross-encoder reranker (Cohere Rerank 3, BGE-reranker-v2, Voyage rerank) scores each candidate against the query as a pair, and returns a re-ordered list. Keeping the top 4–6 typically cuts hallucinations by 40–60%.

Reranking is not optional for anything beyond a 100-document corpus. The cost of one reranker call is orders of magnitude smaller than the cost of a hallucinated LLM answer that a human has to correct downstream.

Users type “how do I refund?” — but your docs say “issuing a refund to a customer via the merchant dashboard.” The embedding gap between those two strings is real. Two techniques close it:

1. HyDE (Hypothetical Document Embeddings) — ask the LLM to draft a hypothetical answer to the query, embed that, and search with the answer-shaped vector.
2. Multi-query fan-out — ask the LLM to generate 3–5 rephrasings, retrieve top-k for each, deduplicate by ID, and rerank the union.

Multi-query is my default. HyDE is cheaper (one extra LLM call vs. three or four retrieval round-trips) but hallucinated answers can send retrieval sideways. Measure both on your golden set before committing.

async def multi_query_retrieve(user_q: str, k: int = 8):
    variants = await llm.generate_variants(user_q, n=4)
    queries = [user_q, *variants]
    hits = await asyncio.gather(*(hybrid_search(q, k=20) for q in queries))
    merged = dedupe_by_id(flatten(hits))
    return await rerank(user_q, merged, top_k=k)

See async Python for AI pipelines for how to keep the fan-out latency under 300ms end-to-end.

The output prompt is the last line of defense against hallucination. Three rules:

1. The model must answer only from the provided chunks.
2. Every claim must cite the chunk ID that supports it, inline as [chunk_id].
3. If no chunk supports an answer, the model must say so — and offer to escalate or ask a clarifying question.

You are a support agent. Answer ONLY from the CONTEXT below.
Cite each claim with its chunk ID like [c17]. If no chunk supports
the answer, say: "I don't see this in our docs — want me to escalate?"

CONTEXT:
[c14] Refunds for annual subscriptions are prorated per unused month.
[c17] Refund requests must be filed within 30 days of the last charge.
[c22] Refunds are issued to the original payment method within 5–7 business days.

USER: How do I get a refund on an annual plan I bought 45 days ago?

Post-process the output to render [c17] as a clickable link to the source chunk. This one UI change is the single biggest driver of user trust I've measured — from 40% “I don't trust this” to under 10%.

You cannot ship RAG without evals. A 30-example golden set — hand-crafted, versioned in Git, and run on every deploy — prevents 90% of regressions. Grow it to 100–500 examples over the first six months.

Each example has:

• query — the user question, verbatim.
• expected_answer — the ideal response, written by a domain expert.
• required_facts — 1–5 atomic facts the answer must contain.
• required_chunks — chunk IDs that must appear in the retrieved set (recall gate).
• forbidden — strings the answer must not contain (e.g. speculation, wrong product names).

Grade with three metrics: retrieval recall (did the required chunks make it into top-k?), answer correctness (LLM-judge against required_facts with a rubric), and groundedness (did every citation actually support its claim?). Read Evaluating LLMs and classical ML for the full evaluation playbook — the tradeoffs between LLM-as-judge, human review, and classical metrics apply directly here.

# Minimal eval loop — run on every PR via CI
def eval_rag(examples: list[Example], pipeline: Callable) -> Report:
    results = []
    for ex in examples:
        answer, cited_chunks = pipeline(ex.query)
        results.append({
            "query": ex.query,
            "recall": len(set(ex.required_chunks) & set(cited_chunks)) / len(ex.required_chunks),
            "facts": llm_judge_facts(answer, ex.required_facts),
            "grounded": all(fact_in_chunks(a, cited_chunks) for a in split_claims(answer)),
            "forbidden": not any(f in answer for f in ex.forbidden),
        })
    return Report.from_results(results)

You cannot improve what you don't measure. Log one row per query with these fields:

• query, rewrites, retrieved_ids, reranked_ids
• final_answer, cited_ids, abstained (bool)
• latency_ms broken down by stage (embed / search / rerank / llm)
• tokens_in, tokens_out, estimated_cost_usd
• user_feedback (thumbs / rating, when available)

Route logs into a warehouse (BigQuery, Snowflake, or plain Postgres) and build one dashboard per failure mode: abstention rate, recall@10 on tagged eval queries, p95 latency by stage, cost per 1k queries. See Observability for AI systems for the specific alerting thresholds I use, and BigQuery cost control that scales if that's where the logs land.

A typical RAG query costs $0.001–$0.02 depending on retrieval width, reranker choice, and generator model. At 100k queries/day, that's $100–$2000/day. Four levers give you 5–10× compression without measurable quality loss:

1. Cache aggressively. Semantic cache (embedding of the query, cosine ≥ 0.95) catches 15–30% of repeat traffic in most support workloads.
2. Route by difficulty. A cheap classifier decides small-model vs. large-model. See the routing pattern in the agents-with-n8n article.
3. Truncate context. Top-4 reranked chunks are usually enough; the marginal chunk after that costs tokens for near-zero recall lift.
4. Batch embed offline. Never embed at query time except for the query itself. Batched embedding is 5–20× cheaper than on-demand.

For a full cost dashboard template (BigQuery + Looker), see the LLM cost dashboard article.

Every failure mode above has bit me in production. The pattern is always the same: the symptom looks like a model problem and the fix is almost always an engineering problem.

1. Week 1, Days 1–2: Ingestion + structural chunking + one embedding model + pgvector index. No reranker yet.
2. Days 3–4: Write the 30-example golden set. Run baseline.
3. Days 5–7: Add hybrid search (RRF), rerun evals, measure lift.
4. Week 2, Days 8–9: Add reranker, tune top-k, rerun evals.
5. Days 10–11: Grounded prompt + citation rendering + abstention.
6. Days 12–14: Observability, cost logging, and the CI eval gate.

By day 14 you have a system that beats every naïve RAG demo by 20+ points on your own data and is ready for real traffic. Everything after is compounding — better chunkers, better rewrites, better routing.