2026-05-10 · Updated 2026-07-22

Evaluating RAG: Metrics for Every Stage of a Production RAG System

Part 1 of the Production RAG series

A RAG system with broken filters can run for months without triggering an operational alert. It still returns answers and meets its latency target, but the answers rely on incomplete evidence. Recall@k against the original gold set exposes the loss; latency and availability dashboards do not.

Evaluation can detect the failure only when each pipeline stage has its own metric. This article maps common failure modes to those metrics, from document parsing through production monitoring.

[!TIP] Want to skip ahead and run code?

The runnable slavadubrov/rag-evals-demo repository applies the metrics to SciFact. make eval runs the suite, and make benchmark compares chunking, embedding, and LLM configurations. Notebooks 00–09 isolate each metric. The demo uses embedded Qdrant, so it does not require Docker.

TL;DR

• Evaluation defines the system. A stage without a metric is a stage that fails silently.
• A useful evaluation stack covers ingestion, retrieval, generation grounding, ontology conformance, and system signals. RAGAS, TruLens, DeepEval, Arize Phoenix, and the TREC 2024 RAG Track give you tooling. They do not choose your metrics for you.
• For metadata- and ontology-grounded RAG, a wrong tag or brittle hard predicate can collapse recall to zero. Standard Recall@k catches the loss when it retains the original gold set. A filter false-exclusion metric identifies the cause, while faithfulness can still look fine because the model faithfully said “I don’t know.”

The sections follow the order of the pipeline. Start with the decision table, then use the later sections as a reference for each stage.

RAG evaluation decision table

Use this table as the starting point before choosing a framework. The right metric depends on the failure mode you are trying to catch, not on the tool name.

For a shorter tool comparison, see Best RAG Evaluation Tools and Metrics in 2026.

Part 1: Define success before the architecture

Draft the eval set before the architecture diagram. It gives each later component choice a measurable target.

You cannot choose between BM25 and dense retrieval, recursive and semantic chunking, or Cohere Rerank and BGE until you know what you are optimizing. “Better answers” is not a metric. An illustrative contract is “faithfulness ≥ 0.85 on a 200-query golden set covering our top three intents, with p95 latency < 1.5s and filter false-exclusion rate < 2%.” Its numbers are placeholders; the important part is that quality, coverage, latency, and filtering have explicit gates.

Define the harness before you write the retrieval code. The first harness will be wrong, and you will revise it. Revising a metric is much cheaper than revising a system you have already shipped.

Three pipeline layers and two run modes

Modern RAG is a pipeline, so evaluation has to be a pipeline. No single number catches every failure mode.

Production evaluation has three pipeline layers. Ingestion evaluation asks whether the corpus and index preserve the source. Query-time evaluation asks whether rewriting, filtering, retrieval, reranking, and context assembly found the right evidence. Answer and production evaluation asks whether the response used that evidence and whether quality holds under live traffic. Collapse the layers into one score and a normalization bug can disappear inside an acceptable answer score.

Those layers describe where a failure happens. Offline and online describe when and against which data the check runs. Offline evaluation uses a fixed dataset with known ground truth; it is reproducible and belongs in component selection, A/B comparisons, and CI gates. Online evaluation scores sampled live traffic and captures regeneration, dwell time, explicit feedback, and real query drift. It is noisier and harder to instrument.

Each pipeline layer can contribute offline and online checks. A fixed ingestion corpus catches parser regressions before release, while freshness and parse-failure monitors cover live updates. A fixed query set measures retrieval before release, while sampled live traces expose production drift. Offline-only misses live change; online-only makes regressions hard to reproduce.

Component-level vs. end-to-end

There are two common mistakes. End-to-end-only evaluation tells you the system is broken, but not where. Component-only evaluation can show every part passing while the full system still fails. The fix is a few headline end-to-end metrics for go/no-go decisions, plus component metrics for diagnosis. Retrieval metrics catch retriever regressions. Generation metrics catch generator regressions. End-to-end answer correctness catches integration failures.

The reference frameworks (opinionated tour)

My default stack is RAGAS for the metric vocabulary, DeepEval for CI gates, Phoenix for production tracing, plus custom code for ontology-specific metrics. You will outgrow whatever you start with. Pick the framework that makes custom metrics easy.

For benchmarks, use BEIR (Thakur et al., NeurIPS 2021) for zero-shot retrieval generalization, MTEB for general embedding quality, MIRACL for multilingual retrieval, and the TREC 2024 RAG Track for end-to-end RAG evaluation.

Part 2: Map evaluation points onto the pipeline

A production RAG system is larger than “embed documents, retrieve chunks, call an LLM.” Every stage between document acquisition and answer delivery can fail.

Each stage in the diagram has at least one metric. A stage with no metric can fail without anyone noticing.

The three lanes match where evidence can be lost. The ingestion lane covers parsing, cleaning, chunking, embedding, and indexing. The query-time lane covers rewriting, filtering, retrieval, reranking, and context assembly. The answer and production lane covers faithfulness, citation verification, user signals, drift, latency, and cost.

Errors compound down the chain. Bad parsing limits chunking. Bad chunking limits retrieval. Bad retrieval limits reranking. Bad reranking limits generation. Faithfulness only measures the final answer, never the upstream cause.

Part 3: Ingestion evaluation

Many production RAG failures start in ingestion. The system works on clean test documents, then fails on real PDFs, scans, tables, and messy corpus pages.

Document acquisition and parsing

What to measure:

• Text extraction completeness: extracted_chars / expected_chars on a labeled sample, computed per document class. There is no canonical package — write a small harness that compares parser output against a hand-cleaned reference. Watch for missing footnotes, headers, captions.

• OCR accuracy: CER (Character Error Rate) and WER (Word Error Rate), the standard speech/OCR metrics:

  $$\text{CER} = \frac{S + D + I}{N}, \qquad \text{WER} = \frac{S_w + D_w + I_w}{N_w}$$

  where $S$, $D$, $I$ are character-level substitutions, deletions, insertions and $N$ is the reference character count (subscript $w$ for the word version). Do not apply one CER boundary to every corpus. Calibrate it by document class and downstream answer loss; printed text, handwriting, and multilingual material have different error profiles. Compute with jiwer (jiwer.cer(refs, hyps), jiwer.wer(refs, hyps)) or HuggingFace evaluate. For evaluation corpora, FUNSD and SROIE are public benchmarks.

  from jiwer import cer, wer
  
  refs = ["Mars has two moons, Phobos and Deimos."]
  hyps = ["Mars has two m00ns, Phobos and Deirnos."]
  
  print(f"CER = {cer(refs, hyps):.3f}")  # CER = 0.105
  print(f"WER = {wer(refs, hyps):.3f}")  # WER = 0.286

• Table extraction fidelity: TEDS (Tree-Edit-Distance-based Similarity) measures how close a predicted HTML table tree is to the reference, normalized by the size of the larger tree. From Zhong et al., 2020 (PubTabNet):

  $$\text{TEDS}(T_a, T_b) = 1 - \frac{\text{EditDist}(T_a, T_b)}{\max(|T_a|, |T_b|)}$$

  TEDS uses both structure (rows, columns, spans) and cell content; TEDS-S strips the content and scores structure only. Reference implementation: PubTabNet’s teds.py (uses apted under the hood). For evaluation corpora, see PubTabNet, FinTabNet, and SciTSR. Naive parsers often fail on tables; benchmark before trusting them.

• Layout / structure preservation: heading order, list integrity, reading order on multi-column PDFs. Use DocLayNet for a labeled benchmark. An off-the-shelf comparison can span an element parser such as unstructured, a PDF library such as pymupdf, and a VLM parser such as docling.

Compare distinct parser families, for example a Tesseract baseline, a VLM-based OCR model, and your vendor candidate. Use a stratified sample of real document classes at a fixed DPI, including clean scans, photos, tables, multilingual text, math, and handwriting. Report CER or WER for each class and TEDS for table pages.

Cleaning and normalization

• Boilerplate removal accuracy: precision/recall against human-labeled boilerplate spans. Aggressive removal kills relevant content; lazy removal pollutes embeddings. Tools to compare: trafilatura, jusText, Resiliparse. Barbaresi (2021) benchmarks these head-to-head.

• Unicode normalization: percent of documents producing identical NFC and NFKC outputs (computed with the stdlib unicodedata.normalize) is a useful drift signal. Mismatches are how zero-width joiners and lookalike characters destroy retrieval recall.

• Language detection accuracy: F1 on a labeled multilingual sample. Critical for multilingual indexes. Use fasttext-langdetect (Facebook’s lid.176), lingua-py, or cld3. FLORES-200 supplies evaluation text across 200 languages, but your production language mix should determine the test slice.

• Deduplication effectiveness (MinHash / LSH): precision/recall of your near-duplicate detector against a hand-labeled set. The underlying idea: estimate Jaccard similarity $J(A, B) = \frac{|A \cap B|}{|A \cup B|}$ between document shingle sets via $k$ random permutation hashes (Broder, 1997) and bucket near-duplicates with LSH banding (Indyk & Motwani, 1998). Sweep the hash count and Jaccard threshold on your corpus. Track false-merge rate (corrupts answers) separately from missed-merge rate (wastes index space). datasketch provides the implementation used below; its parameters are illustrative:

  from datasketch import MinHash, MinHashLSH
  
  def shingles(text: str, k: int = 5) -> set[str]:
      text = text.lower()
      return {text[i:i + k] for i in range(len(text) - k + 1)}
  
  def to_minhash(text: str, num_perm: int = 128) -> MinHash:
      m = MinHash(num_perm=num_perm)
      for s in shingles(text):
          m.update(s.encode("utf-8"))
      return m
  
  docs = {
      "d1": "Mars has two moons, Phobos and Deimos.",
      "d2": "Mars has two moons, Phobos and Deimos!",   # near-dup
      "d3": "Curiosity rover landed on Mars in 2012.",
  }
  
  lsh = MinHashLSH(threshold=0.8, num_perm=128)
  for did, text in docs.items():
      lsh.insert(did, to_minhash(text))
  
  print(sorted(lsh.query(to_minhash(docs["d1"]))))  # ['d1', 'd2']

• PII scrubbing: precision and recall, computed separately per entity type (emails, SSNs, names, addresses). Recall errors create compliance risk; precision errors hurt answer quality. Set the operating point with the legal team. Candidate tools include Microsoft Presidio, scrubadub, or a fine-tuned NER model on a labeled set.

Chunking controls retrieval quality

Chunking can create a multi-point recall gap even when the embedding model stays fixed. In NVIDIA’s 2024 vendor benchmark, page-level chunking produced the highest accuracy and lowest variance for paginated documents. Treat that result as evidence for the tested corpus rather than a universal winner.

Semantic chunking groups adjacent sentences by embedding similarity and cuts at dissimilar boundaries. LangChain’s SemanticChunker and LlamaIndex’s SemanticSplitterNodeParser implement this strategy. It can improve recall over fixed windows when topical boundaries matter.

Recursive character splitting tries paragraph breaks, then sentence breaks, then word breaks until each chunk fits the target size. LangChain’s RecursiveCharacterTextSplitter implements the sequence. Choose candidate window and overlap values that fit your document structure, then let the golden set decide the final values.

Metrics to track:

• Chunk coherence: $\text{coherence} = \overline{\cos(s_i, s_j)}_{\text{within}} - \overline{\cos(s_i, s_j)}_{\text{across boundary}}$, where $s_i$ are sentence embeddings. Healthy chunks are internally similar and at-boundary dissimilar. Compute with sentence-transformers plus scikit-learn’s cosine_similarity.
• Boundary quality: human-labeled “is this a sensible cut?” on a sample, plus a structural check that chunks don’t split tables, lists, or numbered sections.
• Optimal chunk size: sweep token sizes (128, 256, 512, 1024) and plot Recall@k vs. size on your golden set. Pick the knee. Don’t pick whatever the tutorial said.
• Overlap effectiveness: ablate several overlap fractions and measure Recall@k. Stop increasing overlap when the local recall curve flattens or duplication cost outweighs the gain.
• Chunk attribution fidelity: percent of chunks that retain a verifiable source pointer (page number, section anchor, doc ID). Auditability requires this.
• Late vs. early chunking: late chunking (Günther et al., 2024) embeds the full document then segments, preserving global context (reference implementation in jina-embeddings-v3). Contextual Retrieval (Anthropic, 2024) prepends LLM-generated context to each chunk. Both add cost. Benchmark on your corpus before adopting either one.

My opinion: structural chunking (splitting on headings, tables, and sections — implemented by parsers like unstructured.io or by walking the AST your parser already produced) is underused. If your documents have structure, use it before adding similarity heuristics. Recursive character splitting is the baseline; semantic chunking is worth the overhead mainly on unstructured prose.

Metadata extraction and enrichment

• NER precision/recall/F1: per entity type, on a labeled subset. Standard CoNLL/MUC-style. Compute with seqeval (from seqeval.metrics import f1_score) for the BIO/IOB-tag-aware version, or scikit-learn for span-set comparisons. CoNLL-2003 and OntoNotes 5.0 are the canonical reference corpora.
• Relation extraction F1: even more important for ontology-grounded systems. Hand-label a set stratified by relation type and document class. TACRED and DocRED are public benchmarks; candidate implementations include opennre and spaCy relation pipelines.
• Title / heading extraction accuracy: exact-match plus normalized Levenshtein similarity ($1 - \frac{\text{edit\_dist}(a, b)}{\max(|a|, |b|)}$) against ground truth — python-Levenshtein or rapidfuzz give you both in one call.
• Hierarchical metadata preservation: percent of chunks that correctly retain their parent section, parent document, and ancestry path. This is the metric that decides whether your RAG can answer “what does the child of policy X say?” type questions.

Embedding generation

• Model selection benchmarks: MTEB for general capability (nDCG@10 is the headline; the MTEB Python package lets you reproduce the leaderboard locally), BEIR for zero-shot generalization, MIRACL for multilingual. Top retrieval models cluster in a narrow nDCG@10 band, but English MTEB scores poorly predict performance on lower-resource languages.
• Domain-specific evaluation: do not treat a general benchmark rank as a domain result. Size a domain golden set from its coverage matrix and the uncertainty your decision can tolerate. Then re-rank candidate models on it with ranx or pytrec_eval. A domain set can reverse a leaderboard ordering, so publish the dataset slice, retrieval protocol, and confidence interval with the result.
• Embedding drift detection: track distributional KL or model-based drift between a fixed reference window and rolling production embeddings; also measure nearest-neighbor stability for a fixed probe set. evidently and alibi-detect implement model-based and statistical detectors. Evidently’s comparative study is one vendor evaluation; compare methods on known shifts in your own embeddings.
• Multi-vector vs. single-vector: late interaction preserves token-level representations instead of collapsing each document into one vector; ColBERT is the canonical design, with reference implementations in RAGatouille and PyLate. That richer representation increases index and retrieval cost. Compare quality, storage, and latency against a single-vector baseline on the same domain set before adopting it.

Index construction

• Recall@k under approximation: compare the approximate-nearest-neighbour (ANN) index against an exact brute-force baseline at the same k — in FAISS, that’s IndexHNSWFlat (or IndexIVFFlat) vs. IndexFlatIP/IndexFlatL2. Set the acceptable recall loss from your downstream quality budget. The ann-benchmarks project tracks recall–QPS Pareto curves across libraries.
• HNSW tuning: HNSW (Hierarchical Navigable Small World — a layered proximity graph; see Malkov & Yashunin, 2018, implemented in hnswlib, FAISS’s IndexHNSWFlat, and most vector DBs) exposes three knobs: M (graph fan-out), efConstruction (build-time candidate width), efSearch (query-time candidate width). Start from the library’s documented defaults, then sweep the parameters until the recall–latency curve meets your evaluation set’s requirements.
• IVF tuning: IVF (Inverted File index — partition vectors with k-means into nlist cells, then at query time scan the nprobe nearest cells; see FAISS’s IndexIVFFlat and IndexIVFPQ). Sweep nlist and nprobe against exact-search recall and latency. Benchmark filtered queries separately because index families and vector databases implement filter traversal differently.
• Update freshness lag: time from doc commit to retrievability. Track p50 and p99. For systems with regulatory requirements, also track the percent of queries served against stale indexes.

Part 4: Query-time evaluation

The query-time lane contains the metrics that diagnose a retrieval path. Recall@k alone cannot show whether rewriting, filtering, reranking, or context assembly caused the failure.

Query understanding and rewriting

• Query expansion quality: Recall@k uplift on your golden set, expanded query vs. raw. Predefine the minimum useful gain and its uncertainty before testing. If expansion does not clear that local gate, it does not justify its latency and cost. Classical PRF (pseudo-relevance feedback) baselines like RM3 and Bo1 are still useful sanity checks; LLM-based expansion needs to beat them.
• HyDE evaluation: HyDE (Gao et al., 2022) generates a hypothetical answer with the LLM, embeds it, and retrieves against that. It adds generation latency and a new failure surface. Measure Recall@10 separately on in-domain, out-of-domain, and low-confidence slices, then choose whether it belongs in the default path, a fallback, or neither.
• Multi-query generation: Recall@k union of N rewrites vs. single query. Sweep N and choose a point on your recall–latency frontier. Implementations: LangChain’s MultiQueryRetriever, LlamaIndex’s QueryFusionRetriever.
• Intent classification accuracy: standard precision/recall/F1 per intent (compute with sklearn.metrics.classification_report), but the operative metric is routing correctness — does the right downstream pipeline get invoked?
• Adaptive routing: Adaptive-RAG (Jeong et al., NAACL 2024) makes the case that not every query deserves the same retrieval strategy. Track router accuracy as a classification problem against a labeled set of “needs no retrieval / one-shot / iterative.”

Retrieval metrics

These are the baseline metrics. If you do not track them, you cannot tell whether retrieval is improving.

The formulas, for reference (binary relevance with relevant set $R_q$ for query $q$, and $\text{rel}_i = 1$ if the $i$-th retrieved doc is in $R_q$):

For graded relevance, $\text{rel}_i \in \{0, 1, 2, \dots\}$; binary nDCG is the special case used in the code below. MAP is the mean over queries of $\text{AP}_q = \frac{1}{|R_q|}\sum_{i: \text{rel}_i = 1} \text{Precision@}i$. See Manning, Raghavan, Schütze, Introduction to Information Retrieval, chapter 8 for derivations.

For production code, use ranx, pytrec_eval, or ir_measures — they implement the entire TREC metric family and handle graded relevance correctly. Set release targets against a realistic golden set, downstream answer quality, and the cost of a miss. Do not inherit thresholds from a tutorial.

The test harness for these is short. You can run it from a notebook before you’ve even chosen a vector database.

from math import log2
from statistics import mean

# synthetic gold set: query_id -> set of relevant doc ids
gold = {
    "q1": {"d3"},
    "q2": {"d7", "d2"},
    "q3": {"d11"},
    "q4": {"d5"},
}

# ranked retrieval results: query_id -> ranked list of doc ids (top-10)
runs = {
    "q1": ["d8", "d3", "d1", "d4", "d2", "d9", "d6", "d10", "d12", "d13"],
    "q2": ["d2", "d6", "d4", "d7", "d1", "d3", "d8", "d11", "d5", "d9"],
    "q3": ["d11", "d2", "d3", "d4", "d1", "d6", "d7", "d8", "d10", "d12"],
    "q4": ["d1", "d2", "d3", "d6", "d8", "d9", "d10", "d12", "d13", "d14"],
}

def recall_at_k(ranked, gold_set, k):
    if not gold_set:
        return 0.0
    hit = sum(1 for d in ranked[:k] if d in gold_set)
    return hit / len(gold_set)

def reciprocal_rank(ranked, gold_set):
    # MRR contribution per query: 1/rank of the first relevant doc.
    for rank, d in enumerate(ranked, start=1):
        if d in gold_set:
            return 1.0 / rank
    return 0.0

def ndcg_at_k(ranked, gold_set, k):
    # binary relevance: rel ∈ {0, 1}
    gains = [1.0 if d in gold_set else 0.0 for d in ranked[:k]]
    dcg = sum(g / log2(i + 2) for i, g in enumerate(gains))
    # ideal DCG: all gold docs ranked first, capped by k
    n_gold_in_topk = min(k, len(gold_set))
    idcg = sum(1.0 / log2(i + 2) for i in range(n_gold_in_topk))
    return dcg / idcg if idcg else 0.0

K = 5
print(f"Recall@{K}: {mean(recall_at_k(runs[q], gold[q], K) for q in gold):.3f}")
print(f"MRR:       {mean(reciprocal_rank(runs[q], gold[q]) for q in gold):.3f}")
print(f"nDCG@{K}:  {mean(ndcg_at_k(runs[q], gold[q], K) for q in gold):.3f}")
# Recall@5: 0.750
# MRR:       0.625
# nDCG@5:    0.627

That is your retrieval CI gate. Wire it to a coverage-driven fast subset on every PR and run the full golden set on the slower release gate. Block a merge when a preregistered metric crosses its regression budget.

The companion repo pins the exact numbers above (Recall@5 = 0.750, MRR = 0.625, nDCG@5 = 0.627) as a unit test in tests/test_retrieval_metrics.py; notebook 01 sweeps Recall@k / MRR / nDCG over a real SciFact index, and the production-shaped harness lives in evaluation/retrieval.py.

Hybrid retrieval and reciprocal rank fusion

BM25 is a sparse lexical scorer that combines exact-term matching, term weighting, and length normalization. It is available in rank_bm25, Elasticsearch, OpenSearch, and most search engines.

Reciprocal Rank Fusion (Cormack, Clarke, and Buettcher, SIGIR 2009) combines BM25 and dense rankings by position. The original k=60 setting is a useful baseline. RRF is score-agnostic, which avoids the cross-lane normalization required by linear interpolation. With a labeled set large enough to estimate a stable delta, also test a convex combination and tune α.

Hybrid retrieval plus a cross-encoder reranker often improves technical, log-style, and code corpora. The gain can be small on heavily semantic corpora. Measure against the dense-only and sparse-only lanes because a poor fusion configuration can underperform either input.

The implementation fits in a few lines.

from collections import defaultdict

# two retrieval lanes: dense embeddings and BM25.
dense  = ["d3", "d7", "d1", "d4", "d2", "d9", "d10"]
sparse = ["d2", "d3", "d8", "d1", "d11", "d4", "d6"]

def rrf(rankings: list[list[str]], k: int = 60) -> list[tuple[str, float]]:
    """Reciprocal Rank Fusion (Cormack et al., SIGIR 2009).

    score(d) = sum over rankings of 1 / (k + rank(d))
    Score-agnostic: only rank position matters. k=60 is the canonical default.
    """
    scores: dict[str, float] = defaultdict(float)
    for ranking in rankings:
        for rank, doc in enumerate(ranking, start=1):
            scores[doc] += 1.0 / (k + rank)
    return sorted(scores.items(), key=lambda kv: kv[1], reverse=True)

fused = rrf([dense, sparse], k=60)
for doc, score in fused[:5]:
    print(f"{doc}  score={score:.5f}")
# d3  score=0.03252   <- rank 1 dense, rank 2 sparse
# d2  score=0.03178   <- rank 5 dense, rank 1 sparse
# d1  score=0.03150

Note what RRF doesn’t do: it never looks at the raw similarity scores. A dense retriever returning cosine 0.98 and a BM25 lane returning score 17.4 are not directly comparable. If you normalize them with z-scores or min-max scaling, you can end up favoring the lane with the highest variance in that batch.

RRF uses rank only. If a retriever puts a document at position 2, that vote is worth 1 / (60 + 2), regardless of the raw score that produced it.

Hybrid + RRF on SciFact: notebook 02 compares dense vs BM25 vs RRF with per-query deltas. The production-shaped fuser is in retrieval/hybrid_rrf.py; tests/test_rrf.py pins the canonical d3 / d2 / d1 ordering at k=60.

Reranking

• ΔnDCG / ΔMRR: the only honest reranker metric — uplift over no-rerank, on your golden set, at the depth your application actually uses. Compute by running your retrieval metrics with and without the reranker on identical candidate sets.
• Cross-encoder vs. bi-encoder: a bi-encoder embeds query and doc independently (one vector per side) and scores by dot product; a cross-encoder concatenates query+doc and runs a single forward pass that attends jointly across both. Cross-encoders trade a forward pass per candidate for richer query–document interaction. Reference implementation: sentence-transformers CrossEncoder. Benchmark relevance and latency on named hardware, batch size, and candidate depth; do not transfer one model or managed service’s result into another environment.
• Listwise vs. pointwise: pointwise scores each (query, doc) pair independently; listwise scores the whole candidate list jointly so the model can compare candidates. Evaluate both on the same candidate sets. Calibrate any score threshold per model and corpus rather than treating a published example as portable.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-v2-m3")

query = "How do I rotate database credentials in production?"
candidates = [
    "Production database credentials are rotated via Vault every 30 days.",
    "The new logo was unveiled at the all-hands meeting.",
    "To rotate prod DB creds, run the `rotate-secrets` GitHub Action.",
]

scores = reranker.predict([(query, c) for c in candidates])
ranked = sorted(zip(candidates, scores), key=lambda x: -x[1])
for doc, score in ranked:
    print(f"{score:+.3f}  {doc}")

A reranker is a high-leverage candidate for a basic RAG pipeline, not a guaranteed win. Measure its ΔPrecision@1 and ΔnDCG on your golden set, then keep it only if the gain clears its latency and cost budget. Compare that measured gain with smaller retrieval changes before choosing the next optimization.

ΔnDCG and ΔPrecision@1 from a cross-encoder on SciFact: notebook 03; module: retrieval/reranker.py.

Context construction and lost-in-the-middle

This is where many “good retrieval, bad answer” failures come from.

• Context relevance: per-chunk relevance score from RAGAS ContextRelevancy or a cross-encoder, aggregated as mean and as percent of chunks below a threshold.
• Context utilization: of the chunks placed in context, how many were actually cited or used in the answer. Compute as $\frac{|\text{cited chunks}|}{|\text{retrieved chunks}|}$ over a labeled sample. Set the operating threshold from answer quality and token cost rather than using a universal percentage.
• Lost-in-the-middle detection: synthetic eval where you place the gold chunk at positions {first, middle, last} of a long context and measure answer correctness. The U-shaped degradation is real and documented in Liu et al. (TACL 2023). Modern models do better than 2023-era models, but the bias persists. Mitigations: rerank then reorder the top-k so the highest-scored chunk is first or last (LangChain’s LongContextReorder does exactly this), or compress middle chunks aggressively. Measure with a position-stratified eval, not just an aggregate score. A worked, runnable position-stratified eval lives in notebook 06 (module: evaluation/lost_in_middle.py).
• Context compression: report compression ratio (input tokens / output tokens) alongside answer correctness. Tools include LangChain’s ContextualCompressionRetriever and LongLLMLingua. Predefine the largest acceptable correctness loss from the application’s risk and token budget, then reject configurations that cross it.

Part 5: The filter false-exclusion rate

This metric gets its own section because aggregate retrieval scores cannot attribute a miss to the filter.

A hard metadata filter like tenant_id = X AND product = Y AND locale = en-US can drop effective recall to zero. Correctly implemented Recall@k catches the loss because its denominator remains the original relevant-document set. It does not tell you whether the filter, retriever, or ranker caused the miss. Faithfulness can still look fine because it evaluates the answer against the incomplete retrieved context; the model faithfully said “I don’t know.”

The red branch in the tree is the common failure: the right document exists, but the filter removes it before retrieval.

The metric

filter_false_exclusion_rate =
    (# queries where all gold docs were excluded by metadata filter) /
    (# queries with at least one gold doc)

This query-level definition counts catastrophic exclusions: no relevant document survives. For multi-gold queries, standard Recall@k still exposes partial loss; add a per-document exclusion rate if that boundary matters. To compute either rate, you need (a) ground-truth doc IDs for each eval query and (b) instrumentation that logs the filter predicates applied, not just the final results. Set the target from the cost of excluding a valid answer and the confidence interval of your production sample.

Here’s a working implementation. It compares correct standard recall with an invalid evaluator that redefines relevance after filtering.

# A small worked example where hard filters remove relevant documents.
docs = [
    {"id": "d1", "tenant": "acme",   "locale": "en-US"},
    {"id": "d2", "tenant": "acme",   "locale": "en-GB"},
    {"id": "d3", "tenant": "globex", "locale": "en-US"},
    {"id": "d4", "tenant": "acme",   "locale": "en-US"},
    {"id": "d5", "tenant": "acme",   "locale": "de-DE"},
]

queries = [
    # the gold doc lives in en-GB but the dynamic filter forced en-US
    {"qid": "q1", "gold": {"d2"}, "filter": lambda d: d["locale"] == "en-US"},
    # the gold doc is correctly within the tenant filter
    {"qid": "q2", "gold": {"d4"}, "filter": lambda d: d["tenant"] == "acme"},
    # the gold doc is in a different tenant and gets dropped
    {"qid": "q3", "gold": {"d3"}, "filter": lambda d: d["tenant"] == "acme"},
    # the gold doc passes the filter (de-DE locale match)
    {"qid": "q4", "gold": {"d5"}, "filter": lambda d: d["locale"] == "de-DE"},
]

def filter_false_exclusion_rate(queries, docs):
    n_with_gold, n_excluded = 0, 0
    for q in queries:
        if not q["gold"]:
            continue
        n_with_gold += 1
        survivors = {d["id"] for d in docs if q["filter"](d)}
        if not (q["gold"] & survivors):
            n_excluded += 1
    return n_excluded / n_with_gold if n_with_gold else 0.0

rate = filter_false_exclusion_rate(queries, docs)
print(f"filter_false_exclusion_rate = {rate:.2%}")
# filter_false_exclusion_rate = 50.00%

# Correct Recall@k keeps the original gold set as its denominator.
def standard_recall_at_k(queries, docs, k=10):
    recalls = []
    for q in queries:
        survivors = [d for d in docs if q["filter"](d)][:k]
        survivor_ids = {d["id"] for d in survivors}
        recalls.append(len(q["gold"] & survivor_ids) / len(q["gold"]))
    return sum(recalls) / len(recalls) if recalls else 0.0

print(f"standard recall@10 = {standard_recall_at_k(queries, docs):.2%}")
# standard recall@10 = 50.00%

# INVALID: rebuilding the gold set after filtering changes the question.
# It drops queries whose relevant documents did not survive, then scores 100%.
def invalid_recall_over_filtered_gold(queries, docs, k=10):
    recalls = []
    all_doc_ids = {d["id"] for d in docs}
    for q in queries:
        all_survivors = {d["id"] for d in docs if q["filter"](d)}
        filtered_gold = q["gold"] & all_doc_ids & all_survivors
        if not filtered_gold:
            continue
        top_k_ids = set(list(all_survivors)[:k])
        recalls.append(len(filtered_gold & top_k_ids) / len(filtered_gold))
    return sum(recalls) / len(recalls) if recalls else 0.0

invalid = invalid_recall_over_filtered_gold(queries, docs)
print(f"INVALID recall (filtered gold) = {invalid:.2%}")
# INVALID recall (filtered gold) = 100.00%

assert rate == 0.5
assert standard_recall_at_k(queries, docs) == 0.5
assert invalid == 1.0

Half the queries lose their gold doc to the filter, so correct Recall@10 falls to 50%. That score catches the symptom but cannot attribute it. The false-exclusion rate shows that the predicate removed two answers before the retriever ran. The deliberately invalid evaluator reports 100% only because it discards those failures from its gold set. No model can recover a document that was filtered out.

The 50% rate above is reproduced as a unit test in the companion repo: tests/test_filter_exclusion.py::test_50_percent_exclusion_rate. Notebook 04 runs it on SciFact with synthetic metadata so you can watch a real filter zero out recall; the runtime metric (with predicate-precision/recall companion) is in evaluation/filter_exclusion.py.

Companion metric: predicate precision and recall

When filtering is dynamic (for example, an LLM extracts filter predicates from the query), treat the predicate extractor as a classification model and evaluate it as one. Measure predicate precision and recall against a labeled set of (query, correct predicate) pairs. A predicate error rate does not map directly to the same point loss in retrieval recall; measure how often those errors exclude a gold document. Once a hard filter removes the gold document, no amount of reranking helps.

Soft boost vs. hard filter

This metric forces a design decision. Use hard filters when correctness is binary: legal jurisdiction, ACL boundaries, published-versus-draft. Use soft boosts when relevance is graded: locale preference, recency, version. Without exclusion-rate measurement, the wrong choice is hard to see.

The decision rule, measurable:

For each filter predicate F:
  hard_recall_F  = retrieval_recall@k with F as a hard filter
  soft_recall_F  = retrieval_recall@k with F as a +0.X rerank boost
  hard_precision = relevant_in_top_k / k under hard filter
  soft_precision = relevant_in_top_k / k under soft boost
  exclusion_rate = % of queries where the gold doc was filtered out (hard)

Use hard filter only if exclusion_rate < ε AND hard_precision >> soft_precision.
Otherwise prefer soft boost.

Choose ε from the harm of a false exclusion, the benefit of added precision, and the size of the evaluation sample. A dedicated post in this series goes deeper on this trade-off.

Part 6: Generation evaluation

Retrieval metrics tell you the system could answer correctly. They do not tell you it did. Generation metrics cover that gap.

Faithfulness and groundedness

RAGAS faithfulness decomposes the answer into atomic claims (short, self-contained factual statements), then verifies each against the retrieved context via an LLM judge:

The percent of supported claims is the score. The structure is more useful than any single number, because it tells you which claims are unsupported. Production code lives in the ragas package — usage looks like:

from datasets import Dataset
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision

samples = Dataset.from_dict({
    "question": ["How many moons does Mars have?"],
    "answer":   ["Mars has two moons, Phobos and Deimos."],
    "contexts": [["Mars has two moons named Phobos and Deimos."]],
    "ground_truth": ["Mars has two moons."],
})

result = evaluate(samples, metrics=[faithfulness, answer_relevancy, context_precision])
print(result)

Below is the same loop unrolled with a deterministic stand-in judge so you can see the shape end-to-end.

def extract_claims(answer: str) -> list[str]:
    # Production: an LLM call that decomposes the answer.
    # Demo: split on sentence-final punctuation.
    return [c.strip() for c in answer.replace("?", ".").replace("!", ".").split(".") if c.strip()]

def verify_claim(claim: str, context: str) -> bool:
    # Production: an NLI (natural-language inference) model or LLM judge.
    # Demo: a deterministic stand-in so the example runs offline.
    entailed_pairs = {
        "Mars has two moons": True,
        "Phobos and Deimos orbit Mars": True,
        "Mars has a thick atmosphere": False,  # unsupported by context
        "Curiosity landed in 2012": True,
    }
    for k, v in entailed_pairs.items():
        if k.lower() in claim.lower() or claim.lower() in k.lower():
            return v
    words = [w.lower() for w in claim.split() if len(w) > 3]
    return all(w in context.lower() for w in words) if words else False

context = (
    "Mars has two moons, Phobos and Deimos. NASA's Curiosity rover "
    "landed on Mars in 2012."
)
answer = (
    "Mars has two moons. Phobos and Deimos orbit Mars. "
    "Mars has a thick atmosphere. Curiosity landed in 2012."
)

claims = extract_claims(answer)
verdicts = [(c, verify_claim(c, context)) for c in claims]
faithfulness = sum(1 for _, ok in verdicts if ok) / len(verdicts)
for c, ok in verdicts:
    print(f"  [{'✓' if ok else '✗'}] {c}")
print(f"faithfulness = {faithfulness:.2f}")
# faithfulness = 0.75   (one unsupported claim about the atmosphere)

The structure matters. In production, verify_claim becomes an NLI model or an LLM call. The rest of the harness stays the same: extract, verify, aggregate.

End-to-end claim extraction + verification on generated SciFact answers: notebook 05; module: evaluation/faithfulness.py. The repo also runs an HHEM-style cross-family verifier in the same loop so you can see which judge family agrees with which.

A purpose-built alternative to LLM-as-judge is HHEM-2.1-Open (Hughes Hallucination Evaluation Model, Vectara), a classifier fine-tuned for hallucination detection. Its model card documents the checkpoint, default decision boundary, and results on AggreFact and RAGTruth. Treat those as model-card evidence, not a guarantee on your corpus: calibrate the threshold on local labels and compare it with your chosen judge before deployment.

Atomic-fact evaluation

FActScore (Min et al., EMNLP 2023) decomposes long-form generations into atomic facts, retrieves evidence per fact, labels each supported / not-supported, and reports the supported fraction:

Reference implementation: shmsw25/FActScore. It works well for biographies, summaries, and other long-form outputs. Watch out: repetitive trivial facts can inflate the score, and “MontageLie” attacks (true facts in deceptive order) can defeat it. VeriScore handles claims with necessary modifiers; the Core filter helps prevent fact-padding.

Citation accuracy

Track citation precision (cited spans actually support the claim) and citation recall (claims that should be cited, are):

The TREC 2024 RAG Track defines a reproducible support evaluation protocol. Upadhyay et al. (SIGIR 2025) report GPT-4o agreeing with human judges 56% of the time on manual assessment from scratch, rising to 72% with post-editing of LLM predictions. That is useful as a force multiplier under their conditions, not as a replacement for human assessment in high-stakes contexts. For an automated approximation, ALCE (Gao et al., EMNLP 2023) implements citation precision/recall with NLI-based verification.

Answer correctness, completeness, refusal

• Answer correctness vs. ground truth: when you have it, exact match or token-F1 for short-answer tasks (evaluate.load("squad")), semantic similarity for open-ended (bert-score, embedding cosine via sentence-transformers, or RAGAS AnswerCorrectness).
• Completeness via nuggets: a “nugget” is a single atomic piece of information that any correct answer must contain (e.g., for “When was the company founded?” the nuggets might be {year: 1994, founder: Jane Doe}). TREC’s AutoNuggetizer extracts the gold nuggets of a correct answer from a reference, then scores what fraction the system covers — strong correlation with manual evaluation across 21 topics × 45 runs at TREC 2024.
• Refusal behavior: queries with no answer in the corpus should produce abstention, not hallucination. Track abstention precision (refusals that were correct) and abstention recall (out-of-scope queries that triggered refusal). NoMIRACL is the public benchmark; in your own domain, label a slice of out-of-scope queries and track abstention accuracy.

Post-generation verification

The cheapest reliability gains often come from deterministic post-checks, not larger models.

• Entity grounding check: every named entity in the answer must appear in (or be derivable from) the retrieved context. A simple regex + exact-match check (or spaCy’s ents against a normalized context string) catches a surprising fraction of hallucinations.
• Claim verification: extract claims, run NLI against context, fail or flag any below threshold. NLI-as-faithfulness models: cross-encoder/nli-deberta-v3-large, MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli. Adds latency. Worth it for high-stakes domains.
• Self-consistency (Wang et al., ICLR 2023): sample multiple generations at temperature > 0; report agreement rate (e.g., proportion of generations that match the modal answer, or pairwise BERTScore); choose the sample count from the stability–cost curve and flag low-agreement answers for human review.
• Confidence calibration: collect verbalized confidence (“How confident are you, 0–1?”) and compare to actual correctness on the eval set. Plot a calibration curve and report Expected Calibration Error: $\text{ECE} = \sum_{m=1}^{M} \frac{|B_m|}{n} |\text{acc}(B_m) - \text{conf}(B_m)|$, where $B_m$ are confidence bins. Implementations: netcal, torchmetrics.CalibrationError. A model that reports 0.9 confidence should be correct on roughly 90% of comparable cases; measure the gap instead of assuming calibration.

Part 7: Ontology-grounded RAG evaluation

The standard metrics above cover open-corpus RAG. Ontology-grounded systems need more. If your RAG retrieves against a structured ontology, taxonomy, or knowledge graph (products in a catalog, conditions in SNOMED, components in a BOM, security techniques in MITRE ATT&CK), standard RAG metrics are necessary but not sufficient. You also need to measure the ontology layer.

Entity linking accuracy

The first task is mapping a query mention to an ontology entity (“Aspirin” → wikidata:Q18216, “the 737” → aircraft:Boeing_737).

• Mention-level precision/recall/F1: standard, against gold mention spans (compute with seqeval or a span-set comparator).
• Disambiguation accuracy: of correctly-detected mentions, what fraction map to the right entity ID? Public references include ReFinED, REL, and GENRE; benchmarks like AIDA-CoNLL and BELB show that results vary by system and domain.
• NIL handling: precision/recall on “entity not in ontology.” Measure over-linking to near-but-wrong entities separately from correct abstention.

Hierarchy-aware evaluation

Plain accuracy treats “predicted Sedan when truth is Hatchback” the same as “predicted Sedan when truth is Submarine.” Those errors are not equal.

• Hierarchical precision/recall/F1 (Kosmopoulos et al., 2015): credit ancestors and descendants in the ontology DAG. With $\hat{P}_q$ the predicted node plus all its ancestors and $T_q$ the true node plus all its ancestors:

  $$hP = \frac{\sum_q |\hat{P}_q \cap T_q|}{\sum_q |\hat{P}_q|}, \quad hR = \frac{\sum_q |\hat{P}_q \cap T_q|}{\sum_q |T_q|}, \quad hF1 = \frac{2 \cdot hP \cdot hR}{hP + hR}$$

  Implement with networkx on the ontology graph; see hierarchical-classifier-metrics for a reference.

• Wu-Palmer similarity between predicted and gold entity in the taxonomy (Wu & Palmer, 1994):

  $$\text{WuP}(c_1, c_2) = \frac{2 \cdot \text{depth}(\text{LCA}(c_1, c_2))}{\text{depth}(c_1) + \text{depth}(c_2)}$$

  where LCA is the lowest common ancestor in the taxonomy. Available out of the box in NLTK for WordNet (from nltk.corpus import wordnet as wn; wn.synset("car.n.01").wup_similarity(wn.synset("truck.n.01"))); for custom taxonomies, compute LCA with networkx.

• Sibling/parent confusion rate: separately track confusions to siblings, parents, and children — count_sibling / total_errors, count_parent / total_errors, count_descendant / total_errors. Use reviewed examples to test whether sibling errors come from ambiguous mentions or parent errors from over-generalization.

Filter false-exclusion rate (reprise, now critical)

In ontology-grounded systems, hard filters often come from the ontology itself (“only retrieve docs tagged with category X”). The exclusion-rate metric (defined in Part 5) becomes a primary correctness signal. A wrong category prediction can zero out recall; the exclusion rate attributes that loss to the filter.

Constrained generation conformance

When your output must conform to an ontology (every entity name in the answer must be a valid ontology member; every predicate must come from a closed vocabulary), measure:

• Schema validity rate: percent of outputs that parse and validate against the ontology schema. Validate with jsonschema or pydantic. JSONSchemaBench is the public benchmark for general structured output; for ontology-specific schemas, build your own validator.
• Vocabulary conformance: percent of named entities in the output that are valid ontology IDs — a one-line set-membership check against the closed vocabulary.
• Semantic conformance: validity is necessary but insufficient. A syntactically valid output can pick the wrong-but-valid entity. Pair conformance with downstream answer correctness.

Constrained decoding frameworks (Outlines, XGrammar, Guidance, OpenAI Structured Outputs) are designed to enforce schema validity. JSONSchemaBench compares efficiency, coverage, and quality across implementations. Re-run its cases that match your schemas and serving backend because coverage and latency depend on both.

Auditability

For ontology-grounded systems where answers face review:

• Citation completeness: percent of factual claims with at least one verifiable citation.
• Provenance depth: percent of citations that resolve all the way back to a source document with a stable ID, not just a chunk hash.
• Reproducibility rate: re-running the same query at a fixed snapshot returns the same answer. Pin the model version, runtime, decoding configuration, and seed, then set the required repeat rate from the workflow’s auditability needs. Temperature zero alone does not guarantee determinism. A miss can come from generation, the serving runtime, or any upstream stage.

Part 8: System-level evaluation

Holistic answer quality

• LLM-as-judge (Zheng et al., NeurIPS 2023): a scalable model-based evaluation approach. G-Eval (an LLM-judge protocol that has the model generate its own chain-of-thought rubric before scoring) derives a rubric from a natural-language criterion, then scores with log-prob-weighted output. Agreement depends on the judge, task, prompt, and calibration set.
• Pairwise preference: present judge with answer A vs. answer B; record preference. This avoids absolute-score calibration issues. MT-Bench reported GPT-4 judge agreement above 80% with both human preferences and human–human agreement under its benchmark conditions; do not transfer that rate to another domain without calibration.

LLM-as-judge has real biases:

• Position bias: judges prefer the first or second answer regardless of quality. Mitigation: randomize order, or run both orders and average.
• Verbosity bias: judges can confound length with quality. A 2026 controlled study found heterogeneous expansion-pair behavior: three judges preferred longer answers, Claude preferred concise answers, and GPT-4o was approximately neutral. All five performed well on truncation controls. The result is benchmark-bound, so tell your judge how to treat completeness and filler, then report length-controlled performance on your own rubric.
• Self-preference bias: GPT-4 prefers GPT-4 outputs; the bias correlates with output perplexity (judges prefer text that’s familiar to them). Mitigation: use a different judge family from the system being evaluated. Do not use a model to judge itself.

Practical recipe: select a judge on human-labeled calibration data, randomize answer order, mask model identities, and state the length policy in the rubric. Repeat cases only when the added samples materially reduce uncertainty. For high-stakes evaluations, compare judges from different model families and analyze disagreements against human labels.

Schema-Guided Reasoning for judges

Free-form output is one source of variation in judge runs. Two runs against the same answer may organize the rubric differently and produce different scores. Schema-Guided Reasoning (SGR) makes that rubric explicit: define the evaluation stages as a Pydantic schema, then use constrained output through Outlines, XGrammar, vLLM structured outputs, or OpenAI response_format so every run returns the same fields in the same order.

For RAG eval the schema decomposes the judgment into explicit, auditable fields rather than letting the model jump straight to a number:

from pydantic import BaseModel, Field
from typing import Literal

class FaithfulnessJudgment(BaseModel):
    extracted_claims: list[str] = Field(
        description="Atomic factual claims in the answer, one per item."
    )
    supported_claims: list[str] = Field(
        description="Subset of extracted_claims that are entailed by the context."
    )
    unsupported_claims: list[str] = Field(
        description="Subset that is NOT entailed by the context."
    )
    failure_mode: Literal[
        "none", "fabrication", "overgeneralization", "wrong_entity", "stale_fact"
    ]
    score: float = Field(ge=0.0, le=1.0)
    rationale: str

The structured fields make the score recoverable as len(supported) / len(extracted) and show exactly which claims two judges disagreed about. The Pydantic model also makes a rubric change visible as a code diff. Constrained output guarantees the shape, not an unbiased verdict, so position randomization, cross-family judges, and human calibration still apply.

This works for any rubric-based judge, not just faithfulness. Pairwise preference, citation support, and refusal correctness all benefit from the same treatment.

A G-Eval / pairwise / position-bias / cross-family judge harness lives in notebook 07; module: evaluation/llm_judge.py. The benchmark sweep (make benchmark in the repo) wires three frontier-tier models — gpt-5-mini, claude-haiku-4-5, gemini-2.5-flash — into a rotating-judge pairwise A/B so every model judges the other two, surfacing self-preference numerically.

Latency and cost

• p50, p95, p99 at every pipeline stage. Choose the SLO percentile and alert threshold from the user journey, traffic volume, and error budget.
• Time-to-first-token vs. total generation time. Users care about TTFT for streaming UX.
• Stage breakdown: retrieval, reranking, generation, post-processing. Use the trace to locate the tail instead of assuming which stage caused it; record reranker device and batch size when comparing runs.
• Total $/query = embedding + retrieval + rerank + generation + storage amortized. Track p50 and p99; the long tail is where the budget goes.
• Cache hit rates at the embedding cache, retrieval cache, and KV-cache levels. Set separate targets from observed repetition, invalidation policy, and the cost avoided at each layer.

Per-stage p50/p95/p99 with a stage breakdown is built into notebook 08 and the runner at evaluation/latency.py; the benchmark report combines latency with faithfulness in a single matrix you can re-run with make benchmark.

A/B testing

• Unit of randomization: choose the unit from the estimand, carryover, and interference. Use per-user or per-session assignment when repeated exposure can change behavior or create inconsistent UX. Per-query assignment is defensible only when those effects are negligible and the analysis models repeated observations.
• Primary, guardrails, exploratory metrics: pre-register them. Choose the primary measure from the product outcome; satisfaction proxies include thumbs, regenerations, and dwell. Treat latency and cost as guardrails when they constrain the experience.
• Sample size: power-analyze before launching from the minimum effect worth detecting, baseline variance, assignment unit, and stopping rule.

Part 9: Test set construction

A metric is only as good as the test set it runs on. If your golden set covers three intents and production traffic spans twelve, Recall@10 measures only those three intents. Worse, a test set that overfits to easy questions (“What is the company’s refund policy?”) can approve a system that fails on the hard ones (“Refund eligibility for a partial cancellation under the 2023 EU Digital Services Act, billed in EUR, originating in Ireland?”). The aggregate score rises while the system still fails an important part of production traffic.

The same problem hits ground truth. If SMEs labeled the obvious docs but missed the long-tail relevant ones, Recall@k will under-credit a retriever that actually found them. You optimize toward the labels, not toward the truth.

Build the test set around the real query distribution and difficulty first. Then choose metrics that respond to the target failure modes and tune the system against them.

Synthetic query generation

Use an LLM to generate questions from your corpus:

• Per-chunk: “Generate 3 questions a user might ask that this chunk answers.”
• Multi-hop: sample two chunks, generate a question requiring both.
• Adversarial: generate questions with distractor entities, near-duplicate phrasing, ambiguous mentions.

RAGAS has built-in question-type distribution (reasoning, conditional, multi-context). DataMorgana generates configurable synthetic benchmarks across user and question categories. Synthetic data is useful for cold starts and coverage testing. It cannot replace real user queries.

Golden dataset construction

Human-curated data anchors the golden set.

1. Sample real user queries (or simulated ones if pre-launch) stratified by intent.
2. Have SMEs answer each question and identify which doc(s) contain the answer.
3. Size the set from the coverage matrix and the confidence interval needed for release decisions; coverage matters more than a borrowed query count.
4. Re-curate when release cadence, drift signals, domain risk, and annotation capacity justify it.

Adversarial test sets

• Counterfactuals: swap key entities in the query. Does the system retrieve the right chunks for the swapped query?
• Distractors: queries where the corpus contains a plausible-but-wrong answer that should not be retrieved. This is what RGB (Chen et al., AAAI 2024) stress-tests: noise robustness, negative rejection, information integration, and counterfactual robustness.
• Negation and quantifiers: queries with “not,” “except,” and “only.” Dense retrievers often struggle with these.
• Out-of-scope: queries with no answer in the corpus. The system should say “I don’t know,” not hallucinate. NoMIRACL lives here. Evaluate abstention explicitly on your production query types.

Coverage and continuous evaluation

• Build a coverage matrix: query intent × document type × ontology branch. Aim for ≥1 query per cell. Empty cells are unmonitored regions where regressions hide.
• Run a bounded, fast regression subset on every PR and the full suite on a slower schedule.
• Schedule the full golden-set eval from release cadence and evaluation cost; release candidates are a natural gate.
• Schedule drift evaluation from traffic volume, expected change, and risk. Use a rolling production sample and stratify by feedback rather than silently changing the target distribution.

Part 10: Production monitoring

The eval suite you ship describes the system at launch. Production traffic changes after that.

Implicit and explicit feedback

• Click-through / open rate on cited sources (if your UI exposes them).
• Dwell time on the answer.
• Regeneration rate: percent of answers the user re-asks or asks the system to redo. Treat it as one dissatisfaction signal and calibrate it against reviewed conversations.
• Copy / share / export rates — strong positive signal.
• Follow-up patterns: “Are you sure?” or “But what about X?” patterns suggest distrust.
• Thumbs up/down with optional reason categories (wrong, incomplete, off-topic, harmful, slow). Inline edits, when your UI allows them, are the highest-information feedback signal there is.

Drift detection

• Query drift: track query embedding distribution vs. a reference window using KL divergence, MMD, or a model-based detector. Alert on shift, then segment-debug.
• Embedding drift: pin a probe set of fixed documents; periodically re-embed and measure cosine to the original embeddings. Even small drift between provider model versions silently breaks retrieval. Versioned embedding storage (immutable per-version snapshots) is the cheapest mitigation.
• Performance drift: track production-equivalent metrics (regeneration rate by intent) over time. Sudden jumps mean something broke; slow drifts mean the world changed.

Shadow evaluation and human-in-the-loop

Run the candidate system in parallel with production, compare outputs offline, and do not serve them to users. This catches regressions before launch. It costs extra inference, but it has no customer impact.

For human-in-the-loop (HITL) review:

• Sample low-confidence outputs into a review queue.
• Include a random sample of production traffic for blind review; set its rate from traffic volume, risk, and reviewer capacity.
• Weight thumbs-down outputs heavily.
• Use reviewed outputs to extend the golden set.

The minimum guardrail set

Alert on these, in priority order:

1. Faithfulness/HHEM score below threshold on a rolling production sample.
2. p95 latency above SLO.
3. Filter false-exclusion rate above threshold (sample-based).
4. Regeneration rate outside a locally calibrated control band that accounts for window size, traffic, seasonality, and false-alert budget.
5. Cost/query above budget.

If an alert fires without a corresponding code or model change, you likely have drift. If it fires after a change, you likely have a regression. Either way, you get a signal before support tickets arrive.

Caveats

• Targets are local, not universal. Any number labeled illustrative in this guide is an example configuration or worked result, not a release threshold. Calibrate thresholds to your domain, stakes, evaluation-set uncertainty, and user expectations.
• The framework space moves fast. HHEM versions, RAGAS metric names, model cards, and leaderboard order can drift after publication. Recheck the linked source and re-benchmark before committing.
• LLM-as-judge agreement numbers come with asterisks. The 80% GPT-4-vs-human figure is from MT-Bench / Chatbot Arena conditions. On niche domains and adversarial cases, agreement drops sharply. Use judges as a force multiplier, not a replacement for spot-checking.
• Vendor benchmark uplifts are often not independently reproducible. Reproduce on your own data before believing a number, especially for newer rerankers and OCR systems.
• No metric is a substitute for looking at outputs. Schedule blind review of a random production sample according to traffic, risk, and reviewer capacity. The metrics scale that habit; they do not replace it.

Coming up in this series

This was the index. The follow-ups I am planning:

• Soft Boosts vs. Hard Filters: a deep dive on filter false-exclusion rate, with code, real production examples, and a decision framework.
• Chunking Is the Hidden Variable: a controlled experiment across recursive, semantic, late, and structural chunking on three corpora.
• Reranker Selection in 2026: BGE vs. Cohere vs. ZeRank vs. current cross-encoder models, head-to-head on cost, latency, and uplift.
• Ontology-Grounded RAG: An End-to-End Walkthrough: building the full evaluation harness for an entity-grounded retrieval system.
• LLM-as-Judge Without the Self-Preference Trap: practical recipes for unbiased automated evaluation.
• Online Evaluation in Production: instrumentation patterns, alerting policies, and the dashboards that catch real regressions.

References

Frameworks and benchmarks

• Es et al., Ragas: Automated Evaluation of Retrieval Augmented Generation, 2023.
• RAGAS documentation and GitHub.
• Saad-Falcon et al., ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems, NAACL 2024.
• TruLens, DeepEval, Arize Phoenix.
• Thakur et al., BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models, NeurIPS 2021.
• MTEB Leaderboard.
• TREC 2024 RAG Track.
• Pradeep et al., Initial Nugget Evaluation Results for the TREC 2024 RAG Track with the AutoNuggetizer Framework, 2024.

Retrieval and ranking

• Cormack, Clarke, Buettcher, Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods, SIGIR 2009.
• Gao et al., Precise Zero-Shot Dense Retrieval Without Relevance Labels (HyDE), 2022.
• Jeong et al., Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity, NAACL 2024.
• Anthropic, Introducing Contextual Retrieval, September 2024.
• Günther et al., Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models, 2024.
• Sarthi et al., RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval, 2024.

Generation, faithfulness, judges

• Min et al., FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation, EMNLP 2023.
• Liu et al., Lost in the Middle: How Language Models Use Long Contexts, TACL 2023.
• Chen et al., Benchmarking Large Language Models in Retrieval-Augmented Generation (RGB), AAAI 2024.
• Vectara, HHEM-2.1-Open hallucination evaluation model.
• Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS 2023.
• Upadhyay et al., Support Evaluation for the TREC 2024 RAG Track: Comparing Human versus LLM Judges, SIGIR 2025.
• Thakur et al., NoMIRACL: Knowing When You Don’t Know for Robust Multilingual Retrieval-Augmented Generation, 2023.
• Geng et al., JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models, 2025.
• Kosmopoulos et al., Evaluation Measures for Hierarchical Classification: a unified view and novel approaches, 2015.

Drift and production

• Evidently, Embedding drift detection methods compared.

Companion code

• slavadubrov/rag-evals-demo — runnable harness for every metric in this article on the SciFact corpus, plus a chunking × embedding × LLM benchmark sweep. Notebooks 00–09, unit tests that pin the worked examples above, and an embedded-Qdrant index so it runs without Docker.