Semantic Search at Scale: Hybrid, Rerank, and Filtering Done Right

On a demo corpus of 10k documents, vector search looks magical. Push it to 500k or 5M and the failure modes appear: exact identifiers (order numbers, error codes, product SKUs) never match reliably; rare terms get lost in embedding averages; near-duplicates crowd the top-K; and every query returns 'plausible' but not 'correct' results.

The fix is not a better embedding model. It's a stack: lexical retrieval for precision on tokens, dense retrieval for paraphrase, filters for structured constraints, and a reranker to sort out the mess before it hits the LLM.

Run both retrievers, then fuse the results. The simplest and most robust fusion is Reciprocal Rank Fusion:

def rrf(rank_lists, k=60):
    scores = {}
    for ranking in rank_lists:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
    return sorted(scores.items(), key=lambda x: -x[1])

bm25_hits = bm25.search(query, top_k=50)
dense_hits = index.query(embed(query), top_k=50)
fused = rrf([bm25_hits, dense_hits])[:20]

RRF is parameter-light (only k), handles disparate score scales gracefully, and consistently outperforms weighted linear combinations tuned by hand. Ship it as the baseline and only replace it if you have a specific reason.

Every real search has metadata: tenant_id, language, date range, product line. Two ways to apply filters:

• Pre-filter — restrict the search space before ANN. Fast if the filter is selective (returns <10% of docs). Vector indexes like pgvector's HNSW and Qdrant support this well.
• Post-filter — run ANN over all docs, then drop non-matching results. Safer for low-selectivity filters but requires oversampling top_k or you lose recall.

A bi-encoder (what your embedding model does) scores query and doc independently — fast, but the scoring model never sees them together. A cross-encoder scores them jointly and produces much better rankings, at the cost of one forward pass per (query, doc) pair.

The pattern: retrieve top-50 with hybrid search, rerank with a cross-encoder, keep top-5 for the LLM. Modern rerankers (Cohere Rerank, Voyage Rerank, bge-reranker-v2) cost roughly $1 per 1000 rerank calls and add 100-300ms latency. Nearly every RAG system I've shipped in the last 18 months uses one.

import cohere
co = cohere.Client()
reranked = co.rerank(
    query=query,
    documents=[d.text for d in fused[:50]],
    top_n=5,
    model='rerank-english-v3.0',
)
final = [fused[r.index] for r in reranked.results]

You cannot tune what you cannot measure. Build a golden set:

1. Collect 100-300 real queries from logs.
2. For each query, mark the 1-5 documents that are actually relevant. Use two annotators when you can.
3. Score every configuration on Recall@20, MRR, and nDCG@10.
4. Track a leaderboard in a spreadsheet or MLflow; every change to the pipeline gets a row.

A team without a golden set will spend months debating chunk sizes. A team with a golden set will make measurable weekly progress and know exactly when a change hurts.

For a typical 1M-document, 100 QPS system in 2026:

• BM25 (OpenSearch or Postgres tsvector): $80-200/mo compute.
• Vector index (pgvector on managed Postgres, or Qdrant/Weaviate managed): $150-500/mo.
• Reranker API: ~$400/mo at 8M reranks/month (top-50 × 100 QPS × 8h/day × 22d).
• Embeddings for ingestion: one-time backfill (~$50 per 10M chunks with a small model) plus incremental.

Total: under $1500/mo for a production semantic search backing a mid-sized SaaS. The reranker is often the largest line — cache aggressively for repeated queries.