2026-02-08 · Updated 2026-07-15

Search Ranking Stack in 2026: BM25, Embeddings, Cross-Encoders, and LLM Reranking

Search must satisfy both exact and semantic intent. A query for “wireless headphones” should match those words, but the final order may also depend on product quality, user preferences, and availability. No single ranking method handles all of those signals well.

This post builds the stack one stage at a time: BM25 retrieval, dense embeddings, Reciprocal Rank Fusion, cross-encoder reranking, and finally LLM listwise ranking. A working demo benchmarks every stage on the Amazon ESCI product-search dataset.

TL;DR: Build search as measured stages. Start with BM25, add dense retrieval when it improves recall on your queries, fuse only if the two retrievers have complementary errors, and rerank only the candidate set that fits the latency budget. In this laptop-sized ESCI demo, the full pipeline improved NDCG@10 from 0.585 to 0.717. That result is a worked example, not evidence that every production stack needs all five stages or an online LLM.

Choose stages by failure mode

The production stack is a funnel, but the right funnel depends on the query and business surface.

Start with BM25 as the baseline. Add dense retrieval when vocabulary mismatch hurts recall. Add a cross-encoder when the first page has the right candidates in the wrong order. Add an LLM only after you can afford the latency and can evaluate the ranking decisions.

How we got here

The stack is easier to understand as three layers. Lexical retrieval finds exact terms, dense retrieval closes vocabulary gaps, and rerankers compare the strongest candidates in detail.

BM25 and lexical retrieval

For decades, BM25 was the default. It’s a probabilistic model that scores documents by the frequency of query terms in the document, normalized by document length and inverse document frequency (IDF).

BM25 is strong when literal terms carry the intent: error codes, product SKUs, names, and API identifiers. Its main limitation is vocabulary mismatch. A query for “cheap laptop” may miss a document about a “budget notebook computer” when the indexed text provides no bridge between the expressions.

That said, BM25 is a solid baseline. It scores 0.429 average nDCG@10 across the BEIR benchmark’s 18 datasets and still beats some neural models on argumentative retrieval tasks like Touche-2020.

Dense retrieval and embeddings

BERT-style encoders made dense retrieval practical. They map queries and documents into a shared vector space, then rank candidates with a similarity function such as cosine similarity or dot product.

The bi-encoder (or “two-tower”) architecture processes the query and document independently through separate encoder towers, producing fixed-length embeddings. Document vectors can be pre-computed and indexed offline, then retrieved fast via Approximate Nearest Neighbor (ANN) algorithms. Now “cheap laptop” and “budget notebook” land close together in vector space.

Some bi-encoders use a Siamese architecture, as in Sentence-BERT, where both sides share weights. Others use separate query and document towers. Pooling, vector size, similarity function, and training objective are model choices rather than properties of every dense retriever.

These models are trained with contrastive learning, usually with the InfoNCE loss. Given a batch of (query, positive_document) pairs, the objective maximizes sim(query, positive_doc) while minimizing sim(query, negative_docs). Negatives come from other queries’ positives in the same batch (in-batch negatives). A temperature parameter $\tau$ controls how sharp the distribution is: lower values push the model to make harder distinctions between positives and negatives.

Training data often matters more than the embedding dimension. Retrieval models learn from query-positive pairs and carefully selected hard negatives: plausible but non-relevant documents. The later training section shows how SimANS avoids both trivial negatives and likely false negatives.

The cost is the representation bottleneck. Bi-encoders compress all semantic nuance into a single fixed-size vector, so they often miss fine-grained interactions between specific query terms and specific document content.

Cross-encoders and LLMs

Cross-encoders (Nogueira & Cho, 2019) feed the query and document into a Transformer together as a concatenated sequence ([CLS] Query [SEP] Document), so every query token can attend to every document token through full self-attention. That deep interaction picks up nuance that independent encoding misses.

LLM reranking uses a prompted model to compare several candidates at once. RankGPT showed strong zero-shot listwise results with GPT-4 on its evaluated benchmarks, but output stability, cost, and domain fit still need separate testing.

These scores cannot be pre-computed for arbitrary queries, so reranking is placed after retrieval. That cost asymmetry motivates the multi-stage funnel.

The multi-stage funnel

Running an expensive cross-encoder or LLM across millions of documents isn’t viable, so modern search stacks use a funnel. Each stage filters the candidate pool down while model complexity goes up.

A complex model is too slow to score the whole corpus, while a cheap retriever lacks final precision. The funnel uses each model only where its cost is reasonable.

Retrieval sets the ceiling and reranking optimizes within it. If a relevant document doesn’t survive retrieval, no downstream model can recover it.

The demo: a five-stage pipeline

To make this concrete, I built a search-ranking-stack demo that runs a five-stage pipeline on the Amazon ESCI product search benchmark. Each stage gets measured independently so you can see where the gains actually come from.

The pipeline:

1. BM25 sparse retrieval — lexical baseline (rank_bm25)
2. Dense bi-encoder retrieval — semantic candidate generation (all-MiniLM-L6-v2)
3. Hybrid RRF fusion — rank-based fusion of sparse and dense results
4. Cross-encoder reranking — pairwise relevance scores (ms-marco-MiniLM-L-12-v2)
5. LLM listwise reranking — prompted comparison of the final shortlist (Ollama, API, or local model)

Steps 1—3 are the retrieval stage of the funnel (maximize recall); steps 4—5 are the full ranking stage (maximize precision). The demo skips pre-ranking and blending. At ~8,500 documents, you can afford to send all hybrid results directly to reranking.

Quick start

git clone https://github.com/slavadubrov/search-ranking-stack.git
cd search-ranking-stack
uv sync

# Download and sample ESCI dataset (~2.5GB download, ~5MB sample)
uv run download-data

# Run the full pipeline (without LLM reranking)
uv run run-all

# Run with LLM reranking via Ollama
uv run run-all --llm-mode ollama

Dataset and sampling: Amazon ESCI

The demo uses the Amazon Shopping Queries Dataset (ESCI) from KDD Cup 2022 — a real product search benchmark with four-level graded relevance labels:

Graded relevance matters because it lets you use NDCG (Normalized Discounted Cumulative Gain), which separates a “perfect” ranking from a “merely adequate” one. Binary metrics treat both as equally relevant and can’t tell apart different levels of relevance at the same position.

I used the demo’s small_version sample: about 500 queries, 8,500 products, and 12,000 judgments. It is small enough to run on a laptop, but too small and domain-specific to establish a production ranking. Use it to reproduce the stages and inspect failure modes; use held-out, representative queries to make deployment decisions.

Retrieval: hybrid search

The retrieval layer’s job is to maximize recall — cast the widest net possible so nothing relevant slips through.

BM25: the lexical baseline

BM25 scores documents by term overlap with the query, with term frequency saturation and document length normalization:

Where $\text{IDF}(t)$ is the inverse document frequency of term $t$, $tf(t,d)$ is the term frequency in document $d$, $|d|$ is document length, and $\text{avgdl}$ is the average document length across the corpus. Two parameters matter: $k_1$ (typically 1.2—2.0) controls TF saturation — how quickly repeated terms stop adding value — and $b$ (typically 0.75) controls document length normalization.

The implementation is short. Plain whitespace tokenization with rank_bm25:

# src/search_ranking_stack/stages/s01_bm25.py

from rank_bm25 import BM25Okapi

def run_bm25(data: ESCIData, top_k: int = 100):
    doc_ids = list(data.corpus.keys())
    tokenized_corpus = [text.lower().split() for text in data.corpus.values()]

    bm25 = BM25Okapi(tokenized_corpus)

    results = {}
    for query_id, query_text in data.queries.items():
        scores = bm25.get_scores(query_text.lower().split())
        top_indices = np.argsort(scores)[::-1][:top_k]
        results[query_id] = {doc_ids[idx]: float(scores[idx]) for idx in top_indices}

    return results

BM25 lands at Recall@100 of 0.741 — 74% of relevant products show up somewhere in the top 100. Not bad for a purely lexical method, but 26% of relevant items are invisible to every downstream stage.

Dense bi-encoder retrieval

The bi-encoder maps queries and documents independently into a shared embedding space:

# src/search_ranking_stack/stages/s02_dense.py

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

# Encode corpus once, cache to disk
corpus_embeddings = model.encode(
    doc_texts,
    batch_size=128,
    normalize_embeddings=True,  # Cosine sim = dot product
    convert_to_numpy=True,
)

# At query time: encode query, compute dot product
query_embeddings = model.encode(query_texts, normalize_embeddings=True)
similarity_matrix = np.dot(query_embeddings, corpus_embeddings.T)

With normalized embeddings, cosine similarity reduces to a dot product. The demo computes the full query-by-corpus matrix because 8,500 documents fit comfortably in memory; a production corpus would normally use an approximate-nearest-neighbor index. In this sample, all-MiniLM-L6-v2 raises Recall@100 from 0.741 to 0.825.

How bi-encoders learn good representations

Bi-encoder training usually runs in two phases. First, the model gets pre-trained on Natural Language Inference (NLI) and Semantic Textual Similarity (STS) datasets, which teach general-purpose semantic understanding — the model learns that “a cat sits on a mat” and “a feline rests on a rug” should have similar embeddings. Second, it’s fine-tuned on retrieval-specific data like MS MARCO, where it learns that a search query and its relevant passage should sit closer together than the query and irrelevant passages.

The critical ingredient in the second phase is hard negative mining. Random negatives (e.g., a document about cooking paired with a query about headphones) are trivially easy to tell apart — the model learns nothing from them. Instead, you use the current model itself to find documents it ranks highly but that aren’t actually relevant.

The SimANS (Simple Ambiguous Negatives Sampling) approach formalizes this: rank all documents with the current bi-encoder, then exclude easy negatives (ranked too low — the model already handles them) and potential false negatives (ranked too high — they might actually be relevant but unlabeled). The “hard middle ground” produces the maximum learning signal.

# What a training triplet looks like after hard negative mining
training_triplet = {
    "query": "wireless noise canceling headphones",
    "positive": "Sony WH-1000XM5 Wireless Noise Cancelling Headphones",
    "negative": "Sony headphone replacement ear pads",  # Hard negative: same brand, related product, but wrong intent
}
# The bi-encoder must learn that "ear pads" is NOT what the user wants,
# even though it shares many tokens with the positive document.

The contrastive loss function (InfoNCE) ties this together. For each query $q$ with positive document $d^+$ and a set of negative documents $\{d^-_1, \ldots, d^-_n\}$:

Where $\text{sim}(q, d)$ is cosine similarity between query and document embeddings, and $\tau$ is the temperature parameter (typically 0.05—0.1) that controls how sharp the distribution is — lower values make the loss more sensitive to hard negatives. It’s basically a softmax cross-entropy: push the positive pair’s similarity up relative to all negatives. When $\tau$ is small, even slight differences in similarity produce large gradients, which forces the model into finer-grained distinctions.

Serving bi-encoder embeddings at scale

The architectural advantage of a bi-encoder is the offline/online split. Document embeddings are computed at index time and stored in a vector index. At query time, the system encodes the query and searches those stored vectors. Latency depends on the encoder, hardware, index, filters, and recall target, so profile the two steps separately.

In the demo, the math is modest: 8,500 documents $\times$ 384 dimensions $\times$ 4 bytes per float = ~13 MB of embeddings. At production scale the numbers stop being modest: 1 billion documents with 768-dimensional embeddings need ~3 TiB of storage. That’s where quantization (compressing 32-bit floats to 8-bit integers), product quantization (decomposing vectors into subspaces), and SSD-backed indexes like DiskANN come in. The dense-vector indexing section covers the index algorithms.

Why test hybrid retrieval

The two methods often fail differently. BM25 is well suited to proper nouns, product SKUs, and error codes. Dense retrieval can recover vocabulary mismatch such as “cheap laptop” versus “budget notebook computer.” Whether fusion helps depends on how often those complementary cases occur in the target query set.

A common next experiment is hybrid search: run both retrieval methods, then fuse their ranked lists.

Reciprocal rank fusion (RRF)

BM25 and dense retrieval produce scores with different meanings and scales. A linear combination therefore needs calibration and validation whenever the retrievers or corpus change.

Reciprocal Rank Fusion (Cormack et al., 2009) drops raw scores entirely and uses only the rank position:

Here, $k$ is a smoothing constant; 60 is a common starting value. RRF rewards items that rank near the top across input lists without comparing their raw scores. It avoids score-scale calibration, but the retrieval cutoffs, weights, and $k$ still require evaluation.

The implementation:

# src/search_ranking_stack/stages/s03_hybrid_rrf.py

def reciprocal_rank_fusion(ranked_lists, k=60, top_k=100):
    fused_results = {}

    for query_id in all_query_ids:
        rrf_scores = defaultdict(float)

        for results in ranked_lists:
            sorted_docs = sorted(results[query_id].items(),
                                 key=lambda x: x[1], reverse=True)

            for rank, (doc_id, _score) in enumerate(sorted_docs, start=1):
                rrf_scores[doc_id] += 1.0 / (k + rank)

        sorted_rrf = sorted(rrf_scores.items(),
                            key=lambda x: x[1], reverse=True)[:top_k]
        fused_results[query_id] = dict(sorted_rrf)

    return fused_results

Hybrid RRF gets to Recall@100 of 0.842 and NDCG@10 of 0.628 — beating both BM25 (0.585) and Dense (0.611) on their own. Documents only need to rank well in one method to survive fusion.

Cross-encoder reranking

With 100 hybrid candidates per query, you can afford a more expensive model. The cross-encoder processes the query and document together through a single Transformer, with full cross-attention between all tokens.

Token-level interaction

The real difference is in the attention matrix. In a bi-encoder, attention is block-diagonal: query tokens only attend to other query tokens, and document tokens only attend to other document tokens. The two representations never meet at the token level — they only intersect at the end through a dot product. A cross-encoder computes the full attention matrix, where every query token attends to every document token and vice versa. That cross-attention is what unlocks deep token-level interaction.

In a bi-encoder, the query “apple” is encoded before any document is seen. A cross-encoder sees the query and candidate together, so it can use their token-level relationship. This can help with cases such as:

• Negation: “headphones that are not wireless.” A pooled embedding may underweight the negation, while joint encoding gives the model a direct query-document interaction. This is a hypothesis to verify with a targeted slice, not a guarantee.
• Qualification: “laptop under $500.” Joint encoding can relate the constraint to a price in the product text, though structured price filters are safer when the field is available.

The cross-encoder input is formatted as [CLS] query tokens [SEP] document tokens [SEP]. [CLS] is a classification token whose final hidden state is fed through a linear head to produce a single relevance score. Segment embeddings distinguish query tokens from document tokens, and [SEP] marks the boundary between segments.

How cross-encoders are trained

Cross-encoders can learn from (query, document, relevance_label) examples with pointwise, pairwise, or listwise objectives. The pointwise example below uses a single relevance label; it is not the only training design.

# Cross-encoder training data format
training_example = {
    "query": "wireless headphones",
    "document": "Sony WH-1000XM5 Wireless Headphones",
    "label": 1.0,  # Relevant
}
# Forward pass: [CLS] hidden state → Linear layer → sigmoid → score
# Loss: binary cross-entropy between predicted score and label

A common classifier maps the final [CLS] representation to a score. Binary labels may use binary cross-entropy; graded relevance can use regression, ordinal, pairwise, or listwise losses. Choose with held-out ranking metrics rather than assuming one objective is universally better.

Hard negative mining matters even more for cross-encoders than for bi-encoders. Cross-encoders are expensive to train — each training example needs a full forward pass through the concatenated sequence — so you can’t afford to waste compute on trivially easy negatives. The practical recipe: use a bi-encoder to retrieve the top-K candidates for each training query, then pull hard negatives from specific rank ranges (e.g., ranks 10–100). That gives the cross-encoder examples where telling relevant from irrelevant actually requires deep token interaction.

# src/search_ranking_stack/stages/s04_cross_encoder.py

from sentence_transformers import CrossEncoder

model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-12-v2")

def run_cross_encoder(data, hybrid_results, top_k_rerank=50):
    for query_id, query_text in data.queries.items():
        candidates = list(hybrid_results[query_id].items())[:top_k_rerank]

        # Form (query, document) pairs for joint encoding
        pairs = []
        doc_ids = []
        for doc_id, _ in candidates:
            doc_text = data.corpus.get(doc_id, "")[:2048]
            pairs.append([query_text, doc_text])
            doc_ids.append(doc_id)

        # Score all pairs with full cross-attention
        scores = model.predict(pairs, batch_size=64)

        # Rerank by cross-encoder score
        scored_docs = sorted(zip(doc_ids, scores),
                             key=lambda x: x[1], reverse=True)
        reranked_results[query_id] = {
            doc_id: float(score) for doc_id, score in scored_docs
        }

In the recorded demo run, ms-marco-MiniLM-L-12-v2 reranks 50 candidates per query and raises NDCG@10 from 0.628 to 0.645. Measure its latency on the deployment hardware; model, sequence length, batch size, and runtime all affect the result.

The speed-quality tradeoff

Why not use cross-encoders for everything? Because pre-computation is impossible. Bi-encoder document embeddings are query-independent, so you compute them once and store them. A cross-encoder’s output depends on both the query and the document together. The relevance score for “wireless headphones” paired with a Sony product comes from the full cross-attention between those specific tokens. You can’t cache it or reuse it for a different query.

A bi-encoder needs one query encoding plus a vector search over pre-computed document embeddings. A cross-encoder scores every query-document pair in the shortlist, with cost that grows with both candidate count and sequence length. Batching helps, but scoring 100,000 candidates is still the wrong operating point; retrieve first and benchmark the largest shortlist that meets the quality and latency targets.

The rule the demo confirms: Recall@100 stays flat at 0.842 through both reranking stages. Reranking can reorder results but never add documents. Retrieval sets the ceiling.

LLM listwise reranking

The final demo stage uses an LLM for listwise reranking. Instead of scoring each document independently, the model sees the top 10 and returns an ordering. Inspired by RankGPT, the prompt makes relative comparison explicit, but it also introduces context limits, position bias, parsing failures, and run-to-run variance.

The listwise prompt

The prompt template asks the LLM to consider the ESCI relevance hierarchy:

# src/search_ranking_stack/stages/s05_llm_rerank.py

def _create_listwise_prompt(query, documents, max_words=200):
    n = len(documents)

    doc_texts = []
    for i, (doc_id, doc_text) in enumerate(documents, start=1):
        words = doc_text.split()[:max_words]
        doc_texts.append(f"[{i}] {' '.join(words)}")

    return (
        f"I will provide you with {n} product listings, each indicated by "
        f"a numerical identifier [1] to [{n}]. Rank the products based on "
        f'their relevance to the search query: "{query}"\n\n'
        "Consider:\n"
        "- Exact matches should rank highest\n"
        "- Substitutes should rank above complements\n"
        "- Irrelevant products should rank lowest\n\n"
        f"{chr(10).join(doc_texts)}\n\n"
        "Output ONLY a comma-separated list of identifiers: [3], [1], [2], ...\n"
        "Do not explain your reasoning."
    )

Three execution modes

The demo supports three backends for LLM reranking:

Parsing and fallback

LLM outputs are not guaranteed to follow the requested schema, so parsing and a fallback path matter:

def _parse_ranking(output: str, n: int) -> list[int] | None:
    """Parse LLM output to extract ranking order."""
    matches = re.findall(r"\[(\d+)\]", output)

    if not matches:
        return None

    positions = [int(m) - 1 for m in matches]

    # Pad with remaining positions if LLM returned partial output
    if len(positions) < n:
        seen = set(positions)
        for i in range(n):
            if i not in seen:
                positions.append(i)

    return positions[:n]

If parsing fails entirely, the demo falls back to the cross-encoder ordering. A production parser should also reject out-of-range and duplicate identifiers, append omitted candidates in their previous order, log the failure, and compare the fallback rate against a launch threshold.

Results from this ESCI sample

Here are the results from running the full pipeline on ~500 ESCI queries:

Key observations

Hybrid search beats either method on its own. RRF NDCG (0.628) tops both BM25 (0.585) and Dense (0.611). Sparse and dense retrieval have complementary failure modes, and combining them recovers documents that would be missed by either one alone.

Recall is set at retrieval. Recall@100 stays flat at 0.842 through both reranking stages. Rerankers reorder, they don’t add documents. If you want higher recall, fix the retrieval layer.

The prompted LLM produces the largest measured jump in this run. NDCG@10 rises by 0.072 after the LLM stage. The prompt includes the ESCI relevance hierarchy, so the result tests this model-prompt-dataset combination. Repeat runs, confidence intervals, latency, cost, and a held-out domain set are required before attributing the gain to a generally superior reranker.

Dense retrieval beats BM25 on this sample. Inspect query slices before assigning the cause. Vocabulary mismatch is one plausible driver, but the sample construction, tokenizer, model training domain, and corpus fields also affect the comparison.

Evaluation: measuring what matters

The demo uses three complementary metrics. Each one looks at the ranking from a different angle:

NDCG@10 (primary metric)

Normalized Discounted Cumulative Gain measures the quality of the top-10 ranking using graded relevance. It rewards placing highly relevant documents near the top with a logarithmic discount:

NDCG is the only metric that fully uses ESCI’s four-level graded relevance — a system that places an Exact match at position 1 scores higher than one that places a Substitute there. That’s why it’s the primary metric for overall search quality.

MRR@10 (first relevant result)

Mean Reciprocal Rank uses the position of the first result counted as relevant. If it is at position 1, reciprocal rank is 1.0; at position 3, it is 0.333. With graded labels such as ESCI, state the relevance threshold used to turn grades into that binary decision.

Recall@100 (retrieval coverage)

Recall measures what fraction of judged relevant documents appear in the top 100. It is a candidate-ceiling metric for the evaluated judgments: a reranker cannot add a document that retrieval omitted, while incomplete judgments can make the apparent ceiling uncertain.

Indexing dense vectors beyond the demo

Dense embeddings only become useful at scale once you have an Approximate Nearest Neighbor (ANN) index. The demo uses brute-force cosine similarity (which is fine at ~8,500 documents), but production systems need specialized indexes.

HNSW (hierarchical navigable small world)

HNSW builds a multi-layer graph: sparse upper layers navigate broadly, and denser lower layers refine the neighborhood. M controls graph connectivity, while efSearch trades query work for recall. Useful values depend on dimension, distance distribution, filters, implementation, and target recall.

Updates and deletions are an operational consideration because graph indexes may retain tombstones or need background repair. Behavior varies by database. A Qdrant issue, for example, reports degraded quality after a particular heavy-deletion workload. Reproduce the target churn pattern and include compaction or rebuild behavior in the evaluation.

IVF (inverted file)

IVF indexes partition vector space into clusters, then scan the nprobe clusters closest to the query. They can offer a useful memory, build-time, and recall trade-off, especially when combined with compression. Update semantics and performance depend on the implementation rather than the index family alone.

For extreme scale, IVF_RaBitQ (Gao & Long, SIGMOD 2024) compresses floating-point vectors into single-bit representations. In high-dimensional space, a coordinate’s sign (+/-) carries enough angular information for similarity computation.

In one Uber delivery-search case study, reducing a shard-level search parameter from 1,200 to 200 cut reported latency by 34% and CPU by 17% with little measured recall loss. The reusable lesson is to tune the recall-cost curve on production-like traffic, not to copy the value 200.

Optional extensions after the core pipeline

Once retrieval and reranking have separate measurements, several extensions become easier to evaluate without obscuring the core pipeline.

Query understanding

Query expansion and rewriting can address vocabulary mismatch before retrieval. Query2doc generated pseudo-documents and reported BM25 gains on its MS MARCO experiments. Expansion can also introduce the wrong intent, so compare recall and precision on ambiguous, navigational, and exact-identifier query slices.

Practical patterns: abbreviation expansion, entity enrichment, sub-query decomposition for multi-hop reasoning, and RAG-Fusion — generating multiple query variants and combining results via RRF.

LLM-assisted relevance labeling

LLMs can draft relevance labels when human judgments are scarce. TALEC and Pinterest’s relevance-labeling work provide two evaluated designs. An LLM label is still model output: calibrate it against blinded human judgments, inspect disagreement slices, and keep a human gold set for regression tests.

Useful controls include:

• a rubric with concrete relevance boundaries and examples;
• blinded human calibration and periodic rechecks;
• order randomization and repeated judgments for unstable cases;
• model panels where their added cost improves agreement; and
• explicit checks for position and central-tendency bias.

Knowledge distillation

When an LLM teacher adds value but cannot meet serving constraints, distillation is one option:

1. Use a powerful LLM (the teacher) to rerank thousands of training queries
2. Train a small, fast cross-encoder (the student, ~100M–200M parameters) to mimic the LLM’s ranking distribution
3. Compare the student with both teacher and baseline on quality, calibration, and serving cost

InRanker distills MonoT5-3B into 60M and 220M parameter models — a 50x size reduction with competitive performance. The Rank-Without-GPT approach produces 7B open-source listwise rerankers that hit 97% of GPT-4 effectiveness using QLoRA fine-tuning.

Published compression results are starting points, not expected production ratios. Distillation can inherit the teacher’s biases and may lose quality on rare query slices, so keep the original relevance judgments in the evaluation loop.

Personalization and position bias

Generic relevance can only get you so far. A search for “apple” should return iPhones for a tech enthusiast and apple recipes for someone who’s been browsing cooking content.

A common retrieval architecture for personalization uses a two-tower embedding model: the query tower encodes the query and user context, while the item tower encodes items and metadata. The offline/online split supports approximate-nearest-neighbor retrieval; its latency still depends on the encoder, index, filters, and serving system.

Airbnb’s listing embeddings, Pinterest’s OmniSearchSage, and Uber’s two-tower systems show different production designs. Their scale and reported uplifts belong to those systems; the transferable pattern is the offline item tower plus an online query/user tower.

Click data carries position and exposure bias. PAL is one debiasing approach: use position during training, then hold it constant during serving. It is not a universal fix; randomized interventions, inverse-propensity methods, and counterfactual evaluation may be more appropriate for another product.

Domain adaptation with synthetic queries

A common search-strategy mistake is assuming a model trained on general web data (like MS MARCO) will work well on a specialized domain. This is the out-of-domain (OOD) problem.

LLMs can reduce, but not eliminate, the labeled-data bottleneck through Generative Pseudo-Labeling (GPL, InPars):

1. Take your domain-specific document corpus
2. Prompt an LLM to “Generate a search query that this document would answer”
3. Use the synthetic (query, document) pairs to fine-tune your retriever and reranker

Synthetic pairs can help when real queries are scarce, but they reflect the generator and prompt. Deduplicate them, filter implausible queries, and validate on real held-out traffic.

An experiment sequence

Add complexity only when the previous stage exposes a measured failure:

Step 1 (baseline): implement BM25 or the current lexical system and build a judged query set. Record recall, NDCG, latency, and failure slices.

Step 2 (candidate recall): test dense retrieval and fusion only if the baseline misses relevant documents. Tune the candidate cutoff against recall and cost.

Step 3 (ranking precision): add a cross-encoder if the right candidates exist but appear in the wrong order. Choose its shortlist size from a quality-latency curve.

Step 4 (domain fit): fine-tune or distill only after generic models show stable domain-specific failures. Keep real held-out judgments separate from synthetic training data.

Step 5 (optional expensive layer): test listwise or reasoning-based reranking only when its incremental quality survives repeated runs and justifies latency, cost, privacy, and fallback complexity.

Research directions to evaluate separately

Reasoning rerankers and search-using agents are promising, but they answer different questions from the five-stage product-search demo.

Reasoning-based rerankers

Rank1 trains rerankers with reasoning traces and reports strong results on the BRIGHT benchmark. That evidence is relevant to reasoning-heavy retrieval, not a direct prediction for ESCI product search.

For legal or scientific search, compare reasoning rerankers with strong cross-encoder and listwise baselines on expert judgments, citations, latency, and failure consistency.

Agentic search

Search-o1 studies a model that issues additional searches during multi-hop question answering. This is an orchestration problem—query generation, stopping, evidence use, and answer evaluation—rather than another reranking stage. Evaluate it with end-task correctness and citation support, not only retrieval metrics.

Key takeaways

1. Treat the stack as a sequence of experiments. Establish a lexical baseline and a judged query set before adding dense retrieval, fusion, or reranking.

2. Measure candidate recall separately from ranking precision. In this ESCI sample, Recall@100 reaches 0.842 after fusion and stays flat through both reranking stages.

3. Use hybrid retrieval when errors are complementary. RRF improved both Recall@100 and NDCG@10 in the demo, but another corpus may not justify two indexes.

4. Add a cross-encoder when the shortlist is right but the order is wrong. Choose candidate count from a measured quality-latency curve.

5. Treat LLM reranking as an optional final experiment. Its 0.072 NDCG@10 gain here belongs to this prompt, model, sample, and relevance rubric. Repeated runs, strict parsing, fallbacks, and cost belong in the evaluation.

6. Keep the evidence types separate. A paper result, a vendor case study, this laptop demo, and a production A/B test answer different questions.

The complete pipeline code is at github.com/slavadubrov/search-ranking-stack. Clone it, run it, swap in different models and parameters, and see the numbers for yourself.

References

Papers

• Reciprocal Rank Fusion — Cormack et al., 2009
• RankGPT: LLMs as Zero-Shot Listwise Rerankers — Sun et al., EMNLP 2023 Outstanding Paper
• Rank1: Reasoning-Based Reranking — Weller et al., COLM 2025
• SCaLR: Self-Calibrated Listwise Reranking — Self-calibrating listwise reranking framework
• GCCP: Global-Consistent Comparative Pointwise — Addressing calibration in pointwise LLM ranking
• Rank-DistiLLM: Knowledge Distillation for Reranking — Schlatt et al., ECIR 2025
• Query2doc: LLM Query Expansion — Wang et al., EMNLP 2023
• GPL: Generative Pseudo Labeling — Domain adaptation for dense retrieval
• InRanker: Distilled Reranker — 50x size reduction with competitive performance
• Search-o1: Agentic Retrieval — EMNLP 2025
• TALEC: LLM-as-a-Judge for Search — Evaluation framework
• BEIR Benchmark — Thakur et al., NeurIPS 2021
• Sentence-BERT — Reimers & Gurevych, EMNLP 2019
• InfoNCE / CPC — van den Oord et al., 2018
• SimANS: Hard Negative Sampling — Zhou et al., EMNLP 2022
• Passage Reranking with BERT — Nogueira & Cho, 2019
• HNSW — Malkov & Yashunin, 2016
• DiskANN — Subramanya et al., NeurIPS 2019
• RaBitQ — Gao & Long, SIGMOD 2024
• Replacing Judges with Juries — Verga et al., 2024
• RAG-Fusion — Rackauckas, 2024
• InPars — Bonifacio et al., SIGIR 2022
• BRIGHT Benchmark — Su et al., ICLR 2025
• Rank-without-GPT — Zhang et al., ECIR 2025
• Pinterest LLM Search Relevance — Wang et al., 2024
• OmniSearchSage — Agarwal et al., WWW 2024
• PAL: Position-bias Aware Learning — Guo et al., RecSys 2019

Datasets and benchmarks

• Amazon ESCI: Shopping Queries Dataset — KDD Cup 2022
• BEIR: Benchmarking IR — Heterogeneous benchmark for zero-shot evaluation
• MTEB: Massive Text Embedding Benchmark — Embedding model leaderboard
• ESCI Paper — Reddy et al., 2022

Models used in the demo

• all-MiniLM-L6-v2 — 22M parameter bi-encoder
• ms-marco-MiniLM-L-12-v2 — 33M parameter cross-encoder
• Sentence-Transformers — Neural retrieval model framework

Tools and platforms

• rank_bm25 — BM25 implementation in Python
• pytrec_eval — TREC evaluation toolkit
• Elasticsearch — Hybrid search with Retrievers API
• Vespa — Unified search and recommendation engine
• Weaviate — Vector database with hybrid search
• Qdrant — Vector database with multi-stage queries

Industry references

• Airbnb Listing Embeddings — Grbovic & Cheng, KDD 2018
• Uber Delivery Search — Uber Engineering, 2025
• Uber Two-Tower Embeddings — Uber Engineering, 2023
• Elastic Rerank — Elastic, 2024

Demo project

• search-ranking-stack — Working demo with all code from this post