AI Agent Evaluation in Production: Traces to Test Suites

Viacheslav Dubrov

2026-06-10 · Updated 2026-09-06 · 24 min · AI Agents

Article update

Originally published on 10 June 2026. Reviewed and updated on 6 September 2026. The update covers newer agent benchmarks, grader revisions, and evidence on how infrastructure affects scores.

A final answer can say that a refund is complete while its trace shows that verify_identity never ran, issue_refund retried 17 times, or the agent declared success before the database changed. Answer-only grading hides those failures.

For engineers operating tool-using agents in production, the fix is to turn repeatable traces into bounded regression cases: deterministic checks enforce tool order, arguments, loops, and invariants; calibrated judges handle the decisions that require interpretation. The result is a versioned suite that catches the same failure before the next release.

For the short tool comparison, see Best AI Agent Evaluation Tools.

Why agent evals are different

Traditional LLM evals usually score one input-output pair: relevance, faithfulness, correctness, safety, maybe style. Agents add planning, tool calls, retries, and termination checks, and each step is a new place to fail.

Take a refund agent. The transcript can end well while the trace is wrong:

lookup_order -> issue_refund -> final_answer

The output eval passes. A trajectory eval should fail because verify_identity never ran before issue_refund. For tool-using agents, answer-only evals can catch content-quality failures, but they cannot establish that the agent took a valid trajectory or produced the required side effects.

There’s a second problem: errors compound. If a workflow has 20 required steps, each succeeds independently, and every step has the same 95% reliability, its end-to-end success rate lands around 36%:

So the agent can look solid in isolated checks and still fail most full runs. The break is usually somewhere in the middle, and finding it takes component-level visibility, not another look at the answer.

Two research teams put numbers on this.

tau-bench gives an agent airline and retail customer-service tasks. The agent talks to a simulated user, calls APIs, and must follow domain policy. After the conversation, the grader checks whether the database reached the annotated goal state. A plausible transcript with the wrong rows still fails.

Under that grading GPT-4o solved only 35.2% of the airline tasks, and just above 60% of the retail ones. The paper also introduced pass^k: the chance that all k independent trials pass, averaged across tasks.

Retail, the easier split, had pass^8 below 25%. For a randomly selected retail task and eight independent trials, the chance that all eight runs passed was below 25%. A one-run eval cannot measure that consistency.

MAST studies why agents fail. The authors built a 14-mode taxonomy from 150 hand-annotated traces, then applied it across more than 1,600 traces from 7 popular multi-agent frameworks. The taxonomy includes vague role definitions (system design), one agent ignoring what another agent reported (inter-agent misalignment), and declaring success without checking the result (no verification). These failures implicate prompts, orchestration logic, and missing checks in the harness. A stronger base model cannot execute a verification step that was never built, so the evaluation target must include the harness around the model.

The adoption gap

LangChain’s State of Agent Engineering survey (1,340 respondents, fielded in late 2025) suggests that many teams already have the raw material for better evals. It reports that 89% had some observability, 52.4% ran offline evals, and 37.3% ran online evals.

The survey also reports that 57.3% of respondents already have agents in production. When asked what blocks production, 32% named quality and 20% named latency. This is a vendor survey of its respondents, not a census of agent teams, but it exposes a useful gap between trace collection and systematic evaluation.

That leaves teams in an awkward middle state: they can inspect a bad run after the fact, then still ship the same failure twice.

Every diagnosed production failure should leave behind a trace, a label, a dataset row, and a scorer. A repeatable failure belongs in the regression suite.

Pick metrics by failure mode

The right metric depends on the failure mode, not on the framework. The useful split has three levels:

1. Outcome evals answer whether the task succeeded.
2. Trajectory evals answer whether the path was valid, efficient, and policy-compliant.
3. Component evals answer which tool, retriever, sub-agent, or decision step broke.

Each level can run offline on fixed, replayable cases before release or online on sampled production traces after the response. The guardrails section below covers that split in detail. Offline evals can require goldens: stored cases that pair an input with the outcome, tool invariants, and arguments a correct run must produce. Online evals should prefer invariants, distributions, and async checks that stay out of the request path.

Start with deterministic metrics. They repeat for fixed inputs and code, and cost little to run. Their rules can still become stale when tools or policy change, so version the scorer with the specification it checks.

Tool-call correctness

Tool correctness compares the called tools with the expected tools. Pick the strictness deliberately:

• Exact match: the sequence must match exactly. Use this when order is policy, for example lookup_order -> verify_identity -> issue_refund.
• In-order match: required tools must appear in the correct relative order, but extra harmless calls are allowed.
• Any-order match: required tools must appear, but order can vary.

A small local scorer is enough to start:

from collections import Counter

def tool_correctness(called: list[str], expected: list[str], mode: str = "in_order") -> float:
    if mode not in {"exact", "in_order", "any_order"}:
        raise ValueError(f"unknown matching mode: {mode}")
    if mode == "exact":
        return float(called == expected)
    if not expected:
        return 1.0
    if mode == "any_order":
        matched = sum((Counter(called) & Counter(expected)).values())
        return matched / len(expected)

    rows = [[0] * (len(expected) + 1) for _ in range(len(called) + 1)]
    for i, tool in enumerate(called):
        for j, wanted in enumerate(expected):
            if tool == wanted:
                rows[i + 1][j + 1] = rows[i][j] + 1
            else:
                rows[i + 1][j + 1] = max(rows[i][j + 1], rows[i + 1][j])
    return rows[-1][-1] / len(expected)

called = ["lookup_order", "check_refund_policy", "issue_refund"]
expected = ["lookup_order", "verify_identity", "issue_refund"]

print(round(tool_correctness(called, expected, "exact"), 3))     # 0.0
print(round(tool_correctness(called, expected, "in_order"), 3))  # 0.667
assert tool_correctness(["issue_refund"], [], "exact") == 0.0
assert tool_correctness([], [], "exact") == 1.0
assert tool_correctness(["a", "b"], ["a", "a", "b"], "any_order") == 2 / 3
try:
    tool_correctness([], [], "typo")
except ValueError:
    pass
else:
    raise AssertionError("unknown modes must fail")

The in_order score is longest-common-subsequence recall: what fraction of the required sequence survived, in the right order. Notice what it ignores. Junk calls don’t lower it, so an agent can score 1.0 here while making twice the calls it needed. When extra calls cost money or mutate state, track precision alongside it (matched required calls over total calls) and read the two together. Recall catches the missing step; precision catches the wandering. Neither a recall score of 1.0 nor high precision authorizes extra mutations. Check every state-changing call against its permissions, resource, arguments, and required preceding verification. With an empty expected list, only exact mode means “no calls allowed”; the other modes have no positive requirements.

DeepEval’s Tool Correctness metric exposes the same knobs through should_consider_ordering and should_exact_match.

Argument correctness

Calling the right tool with the wrong arguments is often worse than calling the wrong tool because the trace looks normal.

For simple cases, validate JSON schema and exact values. For semantic cases, store expected arguments and grade the deltas:

{
    "trace_id": "tr_2417",
    "input": "Reschedule order A-100 for June 19, 2026.",
    "expected_tools": ["lookup_order", "reschedule_delivery"],
    "expected_arguments": {
        "reschedule_delivery": {
            "order_id": "A-100",
            "date": "2026-06-19"
        }
    }
}

A tool-name metric can’t catch 2026-06-17 where the policy requires 2026-06-19. The dataset has to store arguments too.

For this one-call-per-tool illustration, parameter-match is the fraction of expected (tool, key, value) triples the agent got right. The dictionaries below are valid only when each relevant tool has at most one invocation. Do not build them by overwriting earlier calls with the same name: that would hide a wrong refund followed by a correct one. For repeated calls, retain call IDs and order, match the intended invocation, and validate every mutation separately.

def argument_correctness(called_args: dict, expected_args: dict) -> float:
    total = matched = 0
    for tool, params in expected_args.items():
        for key, want in params.items():
            total += 1
            if key in called_args.get(tool, {}) and called_args[tool][key] == want:
                matched += 1
    return matched / total if total else 1.0

assert argument_correctness({}, {"reschedule_delivery": {"date": None}}) == 0.0
assert argument_correctness({"reschedule_delivery": {}},
                            {"reschedule_delivery": {"date": None}}) == 0.0
assert argument_correctness({"reschedule_delivery": {"date": None}},
                            {"reschedule_delivery": {"date": None}}) == 1.0

Exact equality is right for IDs, enums, and dates already normalized to one format. It’s wrong for free text, floats, and dates in whatever shape the model produced, where == flags a correct answer as wrong. Grade those fields on their own terms: a normalized string match, a date parse, a numeric tolerance. The metric stays the same; the per-field comparator changes.

Efficiency, loops, and dead ends

An agent that completes the task after five redundant tool calls still signals a planning problem and costs more to run.

Cheap signals you should start with:

• Redundant-call rate: identical tool calls with identical arguments repeated more than twice.
• Trace shape anomalies: sudden spikes in depth, tool-call count, token count, latency, or cost.
• Path convergence: how close the run is to the shortest known valid path for the task.
• Termination correctness: whether the agent stopped early, kept working after success, or declared success without the required state change.
• Plan adherence: if the agent writes a plan before acting, check whether the trace followed it. A good plan ignored and a bad plan followed perfectly both fail, for opposite reasons, and the diff between plan and trace tells you which.

Run these before a judge whenever you can. A loop detector is a few lines over the trace. It doesn’t need a model.

Task completion and outcome grading

Judged on the outcome, the question is “did the user get what they asked for?”

Two patterns work best:

• Referenceless task-completion judging: extract the goal from the input and judge whether the trace plus final answer achieved it. This works online because production traffic rarely has golden outputs.
• Environment-state grading: compare the final database rows, files, tickets, bookings, or records to an annotated goal state. This is more robust than transcript matching because agents can find valid paths you didn’t write down.

The second option is better when you can build it. The final state is the contract. The transcript is only evidence.

Two caveats keep this honest. A 2025 audit of agentic benchmarks found that tau-bench grades some tasks purely on the database state. On some tasks, the annotated outcome requires no state change and no specific text. An agent that does nothing can then score a pass: 38% on the airline split and 6.0% on retail, at any k. Anthropic reported an Opus 4.5 run that “failed” a booking task in tau2-bench, the successor benchmark. The agent found a policy loophole that was actually the better outcome for the user. State grading beats transcript matching, but the goal state is still an annotation, and annotations have bugs. Audit the cases that pass too easily, not only the ones that fail.

Benchmark versions and environments change the result

The original tau-bench numbers above explain repeated-trial reliability; they are not the current model leaderboard. The maintained tau-bench repository now presents tau3-bench, adding knowledge retrieval and full-duplex voice. Its July 2026 v1.0.1 grading fix changes banking_knowledge scores: results from earlier versions are not comparable on that domain. Pin the task and grader revisions as well as the model, and re-score saved trajectories when an annotation is corrected.

Choose a benchmark that exercises the deployed interface. Text-only customer-service tests cannot establish interruption handling in a voice agent. Knowledge-retrieval tasks also need the corpus, search configuration, and evidence available at each turn. Borrow these task shapes for local regression cases rather than importing a public leaderboard rank as a release criterion.

The sandbox is part of the test too. Anthropic’s February 2026 infrastructure study found a six-percentage-point Terminal-Bench 2.0 difference between its strict and uncapped resource setups with the same model, harness, and tasks. Extra headroom both reduced infrastructure failures and enabled different solution strategies. Record CPU and RAM guarantees and limits, timeouts, concurrency, network access, and infrastructure-error handling. Report those failures separately without quietly dropping them from the expected-task denominator.

Component evals

Outcome and trajectory metrics tell you the run failed and roughly where. Component evals score one span: was the retrieved chunk relevant, did the sub-agent return the schema its caller expected, did the tool’s own response parse. Attach the score to the span rather than to the run, so “which tool degraded this week” is a query instead of a re-run.

Three checks cover most of it:

• Per-span scoring: run the metric that fits the span type. Retrieval spans get recall and precision against the annotated chunk, sub-agent spans get schema validation plus their own tool-correctness score, tool spans get error rate and latency.
• Tool-result interpretation: feed the agent a correct-but-awkward tool output (an empty list, a partial match, a stale timestamp) and check what it does next. A tool can be right while the agent reads it wrong, and that failure surfaces two steps later.
• Failure attribution: the visible failure is usually downstream of the real one. Attribute to the earliest span whose output was already wrong, not to the step that raised the error.

This is also where the compounding math from the opening pays off. If 20 steps each look fine in isolation, the run can still fail most of the time. Per-span pass rates show which step is running at 95% and which one is running at 70%.

The trace-to-eval flywheel

Mine production failures before brainstorming additional eval cases.

The loop:

1. Capture enough trace evidence to reconstruct the failure, with sensitive content controlled.
2. Label what failed.
3. Group similar failures.
4. Keep representative goldens, including variants that need different outcomes.
5. Version the dataset.
6. Run it in CI.
7. Keep scoring sampled production traces online.

The companion repository trace2evals implements the full loop for a faulty support agent. It captures OpenTelemetry GenAI spans, detects failures with deterministic rules, deduplicates cases into a versioned golden dataset, and reruns each golden in CI. The default backend replaces the model with deterministic rules that re-enact the buggy agent’s decisions, so make demo reproduces the whole loop offline with no API key. Run uv sync --extra live and set an API key, and the same commands drive a real model instead.

This is a teaching pipeline. The revision reviewed on September 6, 2026 still has scorer edge cases, name-based authorization checks, and shared trial state. Its trace adapter expects its own span attributes and message shapes. The corrected examples here do not update that repository or establish production authorization; validate every mutation’s successful, resource-bound authorization and isolate trials before relying on its CI verdicts.

Mine failures with error analysis

Hamel’s field guide shows the workflow: inspect real conversations, take open-ended notes, categorize failures, and build specific tests.

1. Inspect traces and write open-ended notes on what went wrong.
2. Group recurring failures into named categories.
3. Label traces against that taxonomy.
4. Build specific tests for the largest actionable clusters.

Don’t start with labels like reasoning_issue or tool_problem. They’re too vague to test. Use labels like missing_identity_verification, date_argument_mismatch, retried_same_tool_after_429, or stopped_before_database_update. A label that specific tells you exactly what the regression test should assert.

Deduplicate before you promote

The trace-mining loop has a trap: adding every bad trace forever. That creates a dataset that is large, expensive, and narrow. It passes on near-duplicates from March while missing the new shape of the same bug in June.

Group first. Start with one representative golden per cluster, then retain variants with different permissions, arguments, recovery states, or expected outcomes. Similar wording does not make two policy cases equivalent. Store related trace IDs in access-controlled metadata so a reviewer can inspect the evidence later.

If a failure cluster recurs after a fix, the regression case did not generalize. Revisit the cluster and add the missing behavioral variants rather than collecting near-identical transcripts.

Version the dataset

Version datasets the way you version prompts and code. Whenever anything meaningful changes (model, prompt, tool schema, judge prompt, or app behavior), you want to run the same dataset version before and after.

The CI check should pin:

• dataset version
• app version
• prompt version
• judge model
• judge prompt
• evaluator code version
• tool schemas, policy, harness and context-management configuration
• model revision, reasoning and sampling settings
• initial environment fixture and permitted external effects

If any of those moves, your before/after comparison gets muddy. A goldens-v3.json file in git is fine at small scale. Tool-native snapshots in Langfuse, Phoenix, Braintrust, or LangSmith help once the dataset becomes collaborative.

Keep development regressions, judge-calibration examples, held-out validation, and monitoring samples separate. Group related sessions, users, and tasks before splitting so near-duplicates cannot leak across sets. Once a case shapes a prompt or rubric, treat it as development evidence. A mined failure suite tests known regressions; its average does not estimate the production success rate.

Reset mutable state for every trial: files, database rows, caches, and tool fixtures. Isolate credentials and external effects, and keep candidate and baseline budgets equal. Report the number of tasks separately from trials, along with all started attempts, timeouts, crashes, and unscorable outcomes. Compare paired results on the same tasks. These choices follow the clean-trial and outcome-grading approach described in Anthropic’s evaluation guide.

Run evals in CI

A release check must fail the build when a metric crosses its agreed limit. Otherwise the eval suite is only a dashboard.

After restoring the case fixture in an isolated trial, the test should rerun the current agent against the golden input. It shouldn’t merely replay the old failed trace (sketch; the runnable version lives in the companion repository):

@pytest.mark.parametrize("golden", GOLDENS, ids=[item["id"] for item in GOLDENS])
def test_agent_regression(golden: dict) -> None:
    answer, fresh_trace = run_agent_and_capture_trace(golden["input"])

    refired = set(flag_failures(fresh_trace)) & set(golden["failure_modes"])
    assert not refired, f"failure mode regressed: {sorted(refired)}"

    assert tool_correctness(
        called=[call["name"] for call in fresh_trace["tool_calls"]],
        expected=golden["expected_tools"],
        mode=golden.get("tool_match", "in_order"),
    ) >= golden.get("tool_threshold", 1.0)

This distinction is easy to get wrong. The dataset’s job is to catch the next version of the agent repeating an old failure, not to archive the failure itself.

Calibrate the judge before trusting it

LLM-as-judge helps. It’s also easy to fool yourself with.

G-Eval evaluates three meta-evaluation benchmarks. They are SummEval, built from CNN/DailyMail news summarization; Topical-Chat, a knowledge-grounded dialogue benchmark; and QAGS, which tests factual consistency on CNN/DailyMail and XSum summaries. Using GPT-4 as the backbone, G-Eval-4 reached a Spearman correlation of 0.514 with human judgments on SummEval. Its scoring function weights rating levels by token probability ($\text{score} = \sum_i p(s_i)\,s_i$).

The paper estimated GPT-4’s token probabilities by sampling 20 times because that model did not expose them in the experiment. A hosted model may expose no usable logprobs, so keep the rubric but do not imply that you reproduced the paper’s probability weighting. These results compare the paper’s protocol with its NLG baselines on those benchmarks. They support testing an explicit rubric judge, not a general replacement for automatic metrics or a production-agent trajectory benchmark.

MT-Bench showed GPT-4 agreeing with human preferences about as often as humans agree with each other. That result helped make LLM judging mainstream. Later work exposed position, length, and self-preference biases. Judge scores can also shift when the prompt or model version changes.

JudgeBench built response pairs where one answer is objectively wrong across verifiable knowledge, reasoning, math, and code. With a plain judge prompt, GPT-4o scored 50.9%, barely above a coin flip; the paper’s stronger Arena-Hard prompt lifted the same model only to 56.6%. Swapping the model under that stronger prompt matters more: Claude 3.5 Sonnet, the best general-purpose judge tested, reached 64.3%, and o3-mini at high reasoning effort reached 80.9%. Confident but wrong answers stay hard for a judge that does not reason before it grades.

Treat the judge as a measurement instrument: calibrate it against human labels before it grades anything, and recheck it whenever the judge model or prompt moves.

When a judge is required, make the verdict structured. Schema-Guided Reasoning (SGR) gives the verdict a schema for its output shape and inspectability. Structured Outputs or constrained decoding can enforce object shape, required fields, and value constraints for fields such as evidence, passed_criteria, failed_criteria, failure_mode, and score.

Put evidence fields before the score if that makes the record easier to inspect. Field order is presentation, not a reasoning guarantee. A schema-valid verdict can still contain unsupported evidence or an unreliable score. Use calibration against human labels, deterministic validators, and transcript review to test judge reliability. CI can diff a stable JSON object, but that checks inspectability and shape rather than proving that rubric stages were followed.

A structured verdict can also change the cost curve. Treat a cheaper model as a candidate, not an automatic replacement. Run it over the same human-labeled calibration set. Compare its agreement, false-pass rate, and false-fail rate with the larger judge. Use it for routine cases only if it clears the thresholds your application set. Keep the larger judge for disagreements, high-risk cases, or calibration runs.

Default judge hygiene checklist:

1. Prefer binary pass/fail where possible. Five-point scales invite fake precision.
2. Label trajectories covering the actual failure modes before finalizing the rubric. Choose sample size from coverage and the uncertainty the decision can tolerate, then reserve separate cases for validation.
3. Measure judge-human agreement with Cohen’s kappa, a confusion matrix, and positive/negative recall. Kappa measures agreement after accounting for agreement expected by chance; higher is better. A judge that always says “pass” has no useful discrimination, so kappa may be zero or undefined. Decide what to do when it is undefined before using the metric to approve a release.
4. Decompose coarse criteria. “Did the agent verify identity before the refund tool call?” beats “Was the trajectory good?”
5. Emit the verdict through an SGR schema with evidence, failed criteria, failure mode, and score.
6. Compare same- and cross-family judges against held-out human labels; family separation alone does not establish reliability.
7. Measure pairwise order sensitivity. Keep randomization or swapped-order aggregation only if it improves held-out decisions; a 2026 controlled study found that swapping could hurt on adversarial cases.
8. Give no credit for extra text unless it adds correct, relevant, supported content. A longer answer is not a better one.
9. Pin the judge model, prompt, dataset, schema, and app version.
10. Recalibrate after model, prompt, tool, policy, or schema changes.

A panel is another candidate to test. PoLL reported better alignment with human judgments, reduced intra-model bias, and lower cost than its single-GPT-4 baseline across six datasets. Those results belong to its models, tasks, and historical prices. They do not establish that a panel is safer on your task. Compare its false passes, false failures, cost, and disagreement workload with one calibrated judge on held-out labels.

There is no universal kappa threshold that makes a judge suitable for CI. Report the confusion matrix, label counts, false-pass rate among human failures, and false-fail rate among human passes, with uncertainty. Choose release limits from the consequences of those errors. Use review queues when evidence is too weak for automatic acceptance, and retain human authorization for consequential actions where the workflow requires it.

Guardrails block inline, online evals observe afterward

People mix these up because both produce scores. The difference is placement: inline in the request path, before release, or after the response.

Guardrails run inline. They are fast and user-visible. A guardrail can block a tool call, redact PII, reject prompt injection, or force a retry before the response leaves your system. A false positive is a production bug. A false negative is quieter and worse because nothing in the request path reports it. Schema, range, and policy checks are deterministic. Injection and PII detection are classifiers, so treat misses as expected and keep an async eval watching what they let through.

Offline evals run before release. They are reproducible. They check prompts, models, tools, retrievers, and policies against a fixed dataset.

Online evals run after the response, usually on sampled traffic. They can use slower LLM judges because they are not in the latency path. Their job is to detect drift, find new failure clusters, and feed the next offline dataset.

Get the placement wrong and it hurts either way:

• A judge in the request path adds latency and a new source of flakiness.
• A guardrail relegated to async scoring lets policy violations reach users.

For security tests, distinguish attack detection, a prohibited action being attempted, and a harmful effect actually succeeding. Report successful harmful effects per attack trial, with the threat model and attempt budget, alongside legitimate task success and false blocks per benign trial. A detector score alone cannot establish that data stayed private or that a write was prevented. Use isolated targets; Anthropic’s cybersecurity evaluation incident report documents why evaluation effects need containment.

For high-volume systems, score a small sample with a stronger judge and a wider sample with cheaper classifiers. Alert on clusters and confidence bounds, not one noisy point estimate.

Tooling choices

No single tool owns the whole loop. Compare a trace/dataset store and a CI/eval runner separately; one product can cover both, but you do not need to buy both from one vendor.

This is an author snapshot checked on 2026-09-06. Each link is the current documentation I used for the capability claim. Plans, licenses, API keys, provider access, and infrastructure requirements still apply.

The trade-off notes describe where cost comes from, not what it is. Pricing pages move, and vendors count different things: traces, observations, spans, scores, users, retention, or processed data. Recheck live pricing before committing.

Recommendations by constraint:

• Choose Phoenix when self-hosting, privacy, and OTel-compatible tracing are hard requirements and your team can operate the deployment.
• Choose Langfuse when you also need dataset versioning and experiments, and you can operate its storage stack or buy the required add-ons.
• Choose DeepEval when Python/pytest CI pass-fail is the primary contract.
• Choose Inspect AI when the main work is safety or frontier-agent evaluation in configurable sandboxes.
• Choose LangSmith when LangChain/LangGraph integration fits your workflow; use Cloud or account for the Enterprise requirement for BYOC or self-hosted platform hosting.
• Choose Braintrust when managed pull-request feedback and experiment comparison justify an API-key-backed service.
• Choose Promptfoo when prompt or red-team checks are the main regression surface and a trace store is out of scope.

Tool choice is secondary. If production failures don’t become test cases, you’re mostly paying for trace storage.

A practical rollout checklist

Build the evidence pipeline before expanding the metric stack. Start by deciding where the examples will come from.

1. Collect historical runs first. If the agent already exists, pull traces, support tickets, bug reports, thumbs-down sessions, manual QA transcripts, and dogfooding notes before changing the implementation. If the agent does not exist yet, log every prototype and manual test run from day one.

2. Instrument the trace shape. Capture messages, tool calls, arguments, tool outputs, errors, token counts, latency, cost, user feedback, app version, prompt version, model version, tool schema version, and final environment state. Use OpenTelemetry GenAI conventions or OpenInference-style spans if you want portability, and pin the convention version and adapter. Capture content selectively: redact secrets and personal data, restrict access, and set retention before promoting traces into datasets. Use Langfuse, LangSmith, Phoenix, or Braintrust if you want a trace UI and dataset workflow immediately.

3. Turn real failures into seed cases. Read the traces before summarizing them with a model. For each useful failure, store the input, source trace ID, expected state, expected tool invariants, failure mode, severity, and reviewer note. Langfuse can link dataset items back to production traces; LangSmith can create datasets from traced runs. Keep the source link so the case remains auditable.

4. If there is no history, generate cold-start cases. Ask an LLM to draft tasks from product requirements, policies, tool schemas, state machines, and support macros. Cover happy paths and failures such as wrong permissions, missing identity checks, stale tool results, ambiguous dates, retries after rate limits, and contradictory tool output.

5. Do not trust synthetic cases until a human reviews them. Synthetic examples are useful for coverage, not truth. Mark them with source: synthetic and require a reviewer to approve the expected outcome. Run a known-good reference path when possible, and validate generated expectations independently; using another model family does not replace that check.

6. Build a small balanced dataset. Include successes, failures, refusals, boundary cases, long-turn cases, policy-sensitive cases, and valid alternate paths. Do not make the golden “the exact old transcript.” Store what a golden stores above, plus the failure mode that put the case in the suite.

7. Add deterministic checks first. Required tool order where order is policy, required arguments, schema validation, final-state diffs, loop limits, token and latency ceilings, and task-specific invariants should run before any judge.

8. Add one SGR-shaped judge. Use it only for the part that needs interpretation. Calibrate it against human labels and test the chosen rubric on untouched validation cases. If it cannot separate good and bad examples on the calibration set, fix the rubric before wiring it into CI.

9. Wire the loop. Run the small offline suite in CI, run the larger suite before release, score sampled production traffic online, and promote recurring online failure clusters back into the offline dataset.

Your first eval suite will miss cases. Run it anyway, then add recurring failures as cases. A suite that runs every day gives you evidence to improve it.

References

• trace2evals companion demo repository
• LangChain - State of Agent Engineering
• Anthropic - Demystifying Evals for AI Agents
• Hamel Husain - A Field Guide to Rapidly Improving AI Products
• OpenTelemetry semantic conventions for generative AI systems
• DeepEval Tool Correctness
• DeepEval Task Completion
• DeepEval unit testing in CI/CD
• Inspect running evals
• OpenAI Structured model outputs
• Schema-Guided Reasoning: vLLM, xgrammar, and Pydantic
• Phoenix self-hosting
• Langfuse datasets and versioning
• Langfuse experiments in CI/CD
• LangSmith platform setup
• LangSmith - Create and manage datasets programmatically
• Braintrust - Create experiments
• Phoenix LLM evals
• Promptfoo GitHub Action integration
• tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
• Why Do Multi-Agent LLM Systems Fail?
• Establishing Best Practices for Building Rigorous Agentic Benchmarks
• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models
• JudgeBench: A Benchmark for Evaluating LLM-based Judges
• Self-Preference Bias in LLM-as-a-Judge