Harness Engineering for AI Agents: Designing Control Loops

Viacheslav Dubrov

2026-07-22 · Updated 2026-09-06 · 25 min · AI Agents

Article update

Originally published on 22 July 2026. Reviewed and updated on 6 September 2026. The update adds newer harness benchmark evidence and provider-intervention cases, and clarifies what the reported results establish.

An agent can finish its turn while the work is still incomplete. For a coding agent, the useful evidence is the changed artifact and its required test results. A final message saying “done” does not establish either.

The harness is the control code around the reasoning loop. It supplies context, validates and authorizes tool calls, records results, and decides whether the evidence is sufficient to accept the work. The runtime keeps execution and state alive underneath it.

When reviewing a harness, I would ask two questions: what prevents it from accepting incomplete work, and which failures justify its added controls? This article works through acceptance checks, retries, and handoffs, then shows how to compare a control against a fixed baseline. The store examples are fictional; the companion lab is a deterministic simulation, not a measurement of a live agent.

The simplest acceptance check is easy to write for a small research agent like the one this series has been building — a LangGraph agent that fetches market data and writes an analyst report. A hook outside the model validates the report against a schema and checks that it actually contains stock tickers; a malformed report keeps the run open. Twelve lines of ordinary code, and the model does not get to declare its own output well-formed. The repo uses a softer check: a fresh-context evaluator votes, then a human reviews the result. (Part 4 sketches the deterministic version.)

What that example cannot show is the interesting part: what happens when the evidence is ambiguous, when a retry might double-charge someone, or when the work outlives the session that started it. Those need a task with a sharper pass/fail boundary than a research report has. The research agent stays as the acceptance-check example; a small fictional store repository joins it for the retry and handoff cases. The coding task is to lower the threshold for a 10% automatic discount from $100 to $75 in src/checkout.py. The repository has two required checks:

• pytest tests/test_checkout.py verifies the discount calculation.
• pnpm playwright test tests/checkout_discount.spec.ts adds an $80 item in a local test store and checks that the checkout page shows an $8 discount.

The example is a teaching fixture, not a real application or benchmark. Every attempt starts from the same commit and seeded test data. The harness may accept the change only when both commands pass and a durable acceptance record ties both results to a clean committed candidate or a digest of the complete tested snapshot, including relevant untracked files.

The diagram follows the discount change from proposal to evidence. The harness supplies the task and files, checks the proposed edit_file arguments and permissions, and dispatches the accepted call. After the runtime applies the edit, the harness runs the named unit and browser acceptance tests. A failed command goes back to the model as evidence for another turn; two passing commands make the change eligible for acceptance.

What the harness owns

OpenAI’s Codex loop walkthrough describes the basic cycle. The harness assembles a prompt, asks the model for the next action, sends an accepted tool call to the runtime, and appends the result. Then it asks again. That repeats until the harness accepts the result or hands control back to the user.

Implementations may merge several responsibilities into one process. The failure boundaries are still different:

The runtime row covers four things: session, sandbox, checkpoint, and trace. All four store state or confine execution. The model proposes the action, and the reasoning loop chooses the next move. The harness decides whether a proposed call may run and whether the evidence is sufficient to finish, which is why it gets its own article. Part 5 counts the harness alongside those four as one of five primitives you have to place before shipping; this article splits it back out.

When a failure appears, diagnose the boundary that should respond. A poor plan may need better instructions or model reasoning. If edit_file targets a path outside src/, the harness should reject it. A sandbox process that dies before the edit runs belongs to the runtime, which must restart the worker or report the crash.

Where the earlier parts land

The harness row above is doing most of the work in that table, and it is where Parts 2, 3, and 4 end up. Each of them decides one thing about a single turn:

Parts 3 and 4 share step 3, and that overlap is the whole argument for treating them as one program. The same layer of harness code that rejects a malformed argument also rejects a call that is permitted but not yet approved. If validation and authorization run in separate services, preserve the validated arguments across that boundary so the authorization decision applies to the call that will execute.

The split still matters for debugging: a wrong-file edit is a Part 4 path rule, not a Part 2 retrieval problem. A section near the end of this article turns that into a routing table.

OpenAI’s own harness-engineering case study describes a bootable application instance for each worktree. The team also wired browser automation into the agent environment and exposed logs, metrics, and traces.

A task such as “no span in these four critical user journeys exceeds two seconds” became testable because the agent could run the application and query the same signals an engineer would inspect. The case study is product-specific. What transfers is the condition behind the result: the application and its performance signals had to be available inside the agent environment.

Lopopolo, the author of that case study, maintains a field guide for harness engineering. It names the two levers this article pulls: hold the model and coding agent fixed as a black box, and engineer the context and tools around them. His framing also explains why so much of the harness ends up as ordinary code.

An organization’s quality bar, procedures, exception history, and authority relationships sit outside what a general model can know. The harness surfaces them as repository instructions, permission rules, and acceptance checks. Each accepted run can feed its lessons back into those artifacts instead of relying on the next session to rediscover them.

Follow the discount change from proposal to acceptance

For the discount task defined above, the model proposes changing calculate_discount in src/checkout.py. Several things happen before that edit counts as progress:

1. The context builder supplies the task, repository instructions, relevant files, prior tool results, and the current plan.
2. The model proposes an edit_file call with a path and replacement text.
3. The tool boundary (the harness code between proposal and execution) validates the arguments, checks the path against the allowed scope, and asks for approval if the operation needs it.
4. The runtime applies the edit in the sandbox and returns a structured result.
5. The harness runs pytest tests/test_checkout.py, followed by pnpm playwright test tests/checkout_discount.spec.ts, and reads both exit codes. The browser test checks the visible $8 discount on the seeded $80 cart.
6. The harness decides what the results mean. A failed check becomes new context for the next model turn, and a passing run makes the task a completion candidate.
7. A passing result becomes completion evidence only after the harness durably records the command, exit code, tested snapshot, grader, and environment versions; a trace can link to that record.

After step 2, no file has changed. The harness can reject ../../secrets.env, require approval for a destructive command, or stop a run that has exhausted its budget. That is the last cheap moment you get. After the tests run, the harness reads their exit codes itself. The model cannot mark its own edit as passing.

The acceptance record should identify the tested snapshot, both commands and results, and the grader and environment versions; traces can link to that record. Keep required tests outside the agent’s writable scope, or independently approve changes before grading. Any later file edit invalidates the result. These checks implement the stable-environment and bypass-resistant-grader principles in Anthropic’s evaluation guidance. A final done message without those records does not prove that this change passed its required checks.

Decide where each rule is enforced

The requirement that tests/checkout_discount.spec.ts passes belongs in deterministic code, not in the prompt. The harness dispatches the Playwright command to the runtime, reads its exit code, and refuses to end the run while it fails. A prompt can remind the model to run the test. It cannot stop the model from declaring success without evidence.

Other rules fit different layers:

Tool contracts separate proposal from permission

The discount task only needs file edits and test commands. A state-changing API has a different failure mode, so switch examples for this section. Suppose the agent can call create_test_order against a staging order service while setting up test data. This tool is not one of the discount task’s acceptance checks. It is useful here because a timeout can hide whether the service created an order.

The tool boundary needs more than a natural-language description. It needs an explicit tool contract. Part 3 argued for one from the model’s side: clear actions, compact feedback, recoverable errors. The harness needs the same contract for a different reason. It has to decide, without asking the model, whether a call may run and whether a failed call may be repeated. For create_test_order, that means a contract with:

• validated arguments, so malformed input is rejected before execution
• a structured result such as { "order_id": "123", "created": true }, so later checks do not have to parse free-form text
• an effect category that records whether the call only fetches information or changes a file, database record, or external service. It also records whether repeating the call is safe. This label tells the harness whether an automatic retry could duplicate work. The harness can retry get_order_status when the service defines that lookup as read-only. It must not blindly retry create_test_order, because the first call may already have created the order
• a timeout and retry policy, so a lost response does not trigger an unlimited sequence of calls
• a permission rule that says what approval is required. Reading order status may run automatically, while creating an order may require confirmation

The natural-language description is text shown to the model. It might say, “Create a test order for checkout verification.” That sentence helps the model decide when to propose create_test_order. It does not authorize the call. In this example, the harness’s Model Context Protocol (MCP) client validates the arguments, applies its own rules, and checks server trust, approval requirements, and retry safety before dispatching anything. That combines the permission rules and pre-tool checks discussed in Part 4, with one question added: whether a call that already failed may be sent again.

An MCP server publishes tool descriptions and optional behavior annotations to the client. A faulty or malicious server could describe a state-changing tool as harmless. A client that accepted that claim automatically might run or retry create_test_order without approval and create a duplicate. The MCP specification therefore requires clients to treat tool annotations as untrusted unless the server itself is trusted.

The specification does not prescribe one universal trust setting, so you need an explicit trust policy for your deployment; a server cannot make its own annotations trustworthy. That policy decides which metadata may influence permission or retry decisions and which annotations remain advisory only.

Retrying a state-changing call needs replay protection

Part 5 requires a durable operation identity for side effects that can duplicate on retry. The harness is what decides when that key has to carry the weight. create_test_order creates the order but its HTTP response is lost. The harness sees a timeout and cannot tell whether the server completed the request. Repeating the call may create a second order.

Persist an application-owned operation ID before dispatch and bind it to the approved arguments. Reuse it when recovering the same intended order, even if the model generates a new tool-call ID; retain model IDs separately for correlation. Reconcile changed payloads or expired provider deduplication windows instead of blindly resending. Stripe’s contract, for example, permits pruning keys after at least 24 hours.

A status lookup can be retried when the service defines it as read-only. A creation call needs the key: the client attaches a unique request identifier, and the service returns the first result instead of creating another order when it sees that identifier again. Without this protection, the harness should check whether the order exists or ask for a human decision before another attempt. AWS documents this pattern in its idempotent API guidance.

Acceptance needs independent evidence

A successful create_test_order response establishes only that the tool returned data. It does not prove that a coding task passed its tests. If a later browser test depends on the staged order, the harness must validate the response schema and still run that test before accepting the code change.

Some criteria cannot be reduced to an exit code. For a separate visual-design task, a fresh-context evaluator can compare a rendered page or diagram with a written rubric — “fresh-context” meaning a second model session that starts with no history of the run and reads the produced artifacts rather than the transcript. Compare that evaluator with human reviews before you let its result decide whether the task is complete.

A payment-adapter migration needs a handoff

Switch tasks again, but stay in the fictional store repository. The agent now has to migrate checkout from payment adapter v1 to v2. The work spans the checkout handler, payment client, configuration, and tests, so it may outlast one model session — one continuous stretch of model context, ended by a restart or by a deliberate fresh start rather than carried forward.

Before the first session reaches its context limit, it has modified several files, started a local payment sandbox, and left tests/payment_migration.spec.ts failing. That browser acceptance test completes one payment through adapter v2 and verifies the recorded provider ID. A conversation summary can orient the next model session, but it cannot restart the sandbox or prove which files are currently modified.

The next session has to recover three things:

Compaction replaces older messages with a shorter summary so the current session can continue. A progress handoff records what the next session needs: the current branch, changed files, last test command and output, and the next unresolved step.

A handoff file is document memory for the next model session. A checkpoint may already preserve the plan, completed steps, results, and remaining work. Add a handoff when those details are absent or unusable in the next context, and verify files and running services against the live environment.

If the old conversation contains stale assumptions, the harness can start a fresh model session with that handoff and the current workspace. Replacing a crashed worker and restoring its processes is a separate runtime recovery job.

A small documentation edit may need none of these mechanisms. The payment migration needs a handoff if its saved state does not carry usable task progress, because the next model session must reconstruct both the workspace and the task status.

Anthropic’s experiments with long-running coding agents used git history and a progress file between sessions. Anthropic’s later harness-design report separates compaction from a fresh-context handoff and reports that handoffs add orchestration, token use, and wall time, without publishing figures that attribute any of that overhead to the handoff itself.

Use traces to distinguish three failures

The next three rows are illustrative trace sketches, not measured runs or output from the companion lab. Each row shows a different failure and therefore a different harness response.

A transient-looking failure does not make every call safe to retry. The first row is a read-only lookup. The second row is a state-changing request, so the idempotency key and server-side status decide whether another creation attempt is allowed. The third row is not a tool failure at all; the harness has not yet collected the evidence required to accept the discount change.

A chat transcript records what the model saw. It cannot prove whether the order service committed a request before the response disappeared. A trace can supply that evidence only if it includes the relevant server result or status lookup; a client-side timeout alone leaves the outcome unresolved. The durable operation and acceptance records must join the client call, approval decision, operation identity, server result or status lookup, tested snapshot, and acceptance-test result. Traces can expose those links for debugging without becoming the recovery ledger. Those fields tell the harness which of the three paths it is on.

For optional assistance such as planning aids, summaries, and extra evaluators, name the failure and measure whether the component repays its cost. Authorization, isolation, privacy requirements, and mandatory acceptance checks remain in force even if ordinary tasks pass without them. Test those constraints with adversarial cases and explicit invariants; a small success benchmark cannot justify removing them.

Turning those repeated failures into a versioned regression suite is its own job. I wrote that up separately in AI Agent Evaluation in Production.

Measure one change at a time

An ablation measures whether a harness component causes the expected effect by changing or removing that component while the rest of the experiment stays fixed. For example, does editor linting help this model on this task suite?

Use the following protocol:

1. Freeze the model version, task instances, environment, grader, and prompts outside the component under test.
2. Give both variants the same total token, time, and dollar budget.
3. Choose the number of trials or stopping rule before running the comparison.
4. Run the same task instances in both variants. Because model outputs vary, repeat each task several times.
5. Report the mean together with the spread or confidence interval.
6. Count every started trial, including timeouts, policy stops, harness crashes, and evaluator failures.

Success rate alone can hide an expensive component. At minimum, track broken tasks accepted as complete, cost and wall time per completed task, tool errors, duplicate orders, review minutes, and manual permission overrides. Choose the metric that carries the real cost for your product. A two-point rise in completed tasks is a bad trade if it doubles your review queue.

A paired payment-migration experiment makes the progress handoff measurable. Each control/treatment pair starts from the same repository commit and seeded checkpoint, with the same model, task, grader, and total budget. The handoff is the only switch. The primary metric counts duplicate tool actions after resume: an action is duplicate when its operation and artifact match a step that the previous session had already completed.

The SWE-agent paper fixes GPT-4 Turbo on the 300-task SWE-bench Lite split and reports 18.0% resolved with its full interface, compared with 11.0% for a shell-only agent given a worked demonstration and 7.3% for the same agent without one. The paper’s headline 10.7-point gap is measured against that 7.3% baseline; Part 3 works the same three numbers from the interface-design side. The paper also changed individual interface features:

These numbers belong to that model, benchmark, and $4 per-task cap. The three rows below the reference are the useful one-feature tests: each changed an interface feature while the model and evaluation setup stayed fixed.

LangChain published a broader fixed-model comparison for deepagents-cli. It reports a Terminal-Bench 2.0 increase from 52.8% to 66.5% with gpt-5.2-codex fixed while its team changed the system prompt, tools, and middleware. The post bundles several changes and omits a confidence interval, fixed-total-budget comparison, and per-change ablation table. That result cannot identify which change helped. Model names in this section are the ones each study fixed at the time it ran; the protocol is what transfers, not the model list.

A newer comparison shows why the API configuration belongs in the frozen baseline. In its July 29, 2026 ARC-AGI-3 report, OpenAI reports that GPT-5.6 Sol’s public-set score rose from 13.3% to 38.3% when its harness retained reasoning and used compaction instead of dropping reasoning and truncating history. The metric is Relative Human Action Efficiency, not the fraction of tasks solved. This is a vendor-reported, bundled comparison; it does not isolate the two settings or establish a production effect size. For an upgrade, record the API, reasoning retention, compaction policy, and budgets alongside the model ID. Otherwise an apparent model regression may be a missing capability in the adapter.

Include provider intervention in the failure suite. A misalignment_policy_violation must reach a stop-and-review path even after streamed output; it is not a transient retry case. Part 4 covers its API-dependent scope. Test that the harness stops dispatch and records already-completed effects.

Anthropic’s long-running application report is a qualitative, product-specific case study rather than a controlled benchmark. The application is RetroForge, a 2D retro game maker; in Sprint 3 the harness evaluator checked 27 criteria covering its level editor. The work began on earlier Opus models, and when Opus 4.6 shipped the team removed harness components one at a time to see which ones the newer model had made redundant. It reports that evaluator calls became overhead on tasks Opus 4.6 could complete reliably alone, but still helped near the model’s edge. The example is a reason to revalidate old scaffolding when the model changes; it does not estimate a general effect size.

Keep the harness editable after it earns its place

Ablation keeps a harness small, but its code can still outlive the model it was tuned for. A request such as “mask secrets in every capture path” names a behavior, not a file. In a production harness, that behavior may span execution stages and shared state. Before you can change it safely, you have to find every implementation site — and so does the coding agent you delegate it to.

A research-stage option is a 2026 preprint from Wang et al., the Harness Handbook, calls that search behavior localization. The handbook builds a behavior-centric map from the harness codebase. Static analysis, which needs no model calls, extracts a program graph, and an LLM then organizes its units into execution stages.

The maintainer or coding agent starts with a system overview, opens the relevant execution stage, and descends to source-grounded entries for a function or file. A state register records where shared state is written and read between stages. This hierarchy keeps the overview small while preserving a path to source.

Freshness is a separate rule. The map is a navigation aid; live source establishes behavior. Every locator must resolve against the live repository. The handbook freezes stale entries instead of guessing, and each non-empty diff resynchronizes the entries it affects.

The diagram compresses the modification loop: a behavior-only request descends the handbook’s levels, every candidate locator is verified against the live repository before the plan is written, and each applied diff resynchronizes the map.

The Handbook evaluation compares matched arms on 30 requests per repository. It does not establish equal total budgets, repeated stochastic trials, or uncertainty estimates, and its reported comparisons exclude missing outputs and planner errors. It therefore illustrates part of the protocol above, not all of it. It covers two open-source harnesses: Terminus-2 (six Python files) and the Codex monorepo (2,267 Rust files). In each, a read-only planner powered by DeepSeek-V4-Pro either explored the repository directly or routed through the handbook. Requests, repository, tool permissions, and decoding were identical in both arms. Three judges (GPT-5.5, Opus 4.8, DeepSeek-V4-Pro) scored each edit plan on localization, scope control, and reasoning — note that one judge is the same model that produced the plans. A win means one arm’s 0–100 quality score exceeded the other’s by at least three points; otherwise that judge–request comparison was a tie. The reported rate is wins divided by valid judge–request comparisons:

The handbook-assisted planner won more often and used fewer planner tokens in both repositories. The conditions stay attached to that result: three LLM judges scored edit plans produced by one planner model on two harnesses. The study evaluated plans, not executed diffs or production defect rates.

Try the method in the companion lab

The harness-demo project at commit 517353f3 is a small, deterministic exercise with 12 generic synthetic tasks covering code changes such as fix-parser-edge-case, split-large-module, and wire-browser-test. It does not implement the fictional store repository.

Each task fixture declares a difficulty plus four boolean conditions: a flaky tool, lost progress, a missed implementation gap, and ambiguous completion. The simulator derives a fifth condition for difficult tasks that also need a progress file: without context_reset, compaction preserves stale assumptions. A deterministic grader marks a task as passed only when the selected configuration handles every applicable condition. No model or external service runs.

The commands answer different questions:

• make check runs Ruff and seven unit tests, including the validator that rejects any ablation pair that changes more than one component.
• make run prints a cumulative teaching matrix, then five valid leave-one-component-out comparisons.
• make failures names the unhandled condition for every failed task. The full harness should end with all synthetic tasks pass.

make check
make run
make failures

The causal section of make run looks like this:

component                 control  treatment  delta
retry_policy              8/12     12/12       +4
progress_handoff          7/12     12/12       +5
evaluator                 8/12     12/12       +4
fail_closed_acceptance    7/12     12/12       +5
context_reset            10/12     12/12       +2

For each row, the control is the full configuration with one component removed; the treatment restores only that component. The earlier cumulative matrix is useful for orientation, but some of its adjacent rows add several components at once and therefore cannot identify a cause.

The lab validates every declared pair before running it. Its regression tests also include an intentionally invalid pair that changes retry policy and evaluator together; the validator rejects it.

The lab compares all five component fields when it validates a pair. This runnable excerpt shows the same guard on one valid progress-handoff pair:

from dataclasses import dataclass, fields

@dataclass(frozen=True)
class Config:
    progress_handoff: bool = False
    evaluator: bool = False
    retry_policy: bool = False
    fail_closed_acceptance: bool = False
    context_reset: bool = False

def changed_components(control: Config, treatment: Config) -> tuple[str, ...]:
    return tuple(
        field.name
        for field in fields(control)
        if getattr(control, field.name) != getattr(treatment, field.name)
    )

control = Config(progress_handoff=False, evaluator=True, retry_policy=True)
treatment = Config(progress_handoff=True, evaluator=True, retry_policy=True)
assert changed_components(control, treatment) == ("progress_handoff",)

Which layer to open when a run goes wrong

The series ran from the reasoning loop outward. Start with the first failure you see, then investigate the component that owns that job. A run can involve more than one component:

Four rows point to harness code, while row 5 points to the runtime. Instructions can influence behavior, but they cannot replace a permission check, durable checkpoint, or acceptance test.

Start with one loop and one acceptance check

I would start a coding-agent harness with one capable model, repository instructions, a few narrow tools, a sandbox, and one explicit acceptance test. I would record tool calls, results, costs, and that final test in one trace so the first useful failures are visible without reconstructing them from terminal logs and chat transcripts. This is a proposed baseline, not evidence from a deployed system.

From there, add only what a trace justifies. Record who maintains each component, how many tokens or seconds it adds, and which regression test would justify removing it after a model upgrade.

Six months later, someone who sees progress_handoff=True should be able to find the failed traces that justified it and the regression cases that still keep it there. The traces explain why the component exists; a current behavior map explains where to touch it.

If you arrived here from a search, the five preceding articles built a system around a reasoning loop:

1. The loop chooses the next move.
2. Memory supplies context, and a real Postgres checkpoint store preserves it.
3. Tool contracts define actions and the result shapes that later checks can read.
4. Security adds the deny hook and stop-hook validator. Both remain sketches in the example, but they mark the control points.
5. The runtime keeps the process alive across sessions and failures.

The series also added an optional MCP server surface and an evaluator node that checks the draft report before a human sees it. Routing the worker through a credential-holding proxy remains a proposed extension. Those are ordinary pieces of code around a model call. The router is harness code for the same reason: it chooses the reasoning pattern before the reasoning loop starts.

For the next optional assistance component, keep the failed trace, the acceptance rule, and the comparison with that component disabled together. Leave it out if you cannot identify its benefit. Required security and acceptance constraints do not depend on that comparison.

References

• OpenAI, Unrolling the Codex agent loop.
• OpenAI, Harness engineering: leveraging Codex in an agent-first world.
• Lopopolo, Harness engineering: anthology, field guide, and agent context bundle.
• Anthropic Engineering, Effective harnesses for long-running agents.
• Anthropic Engineering, Harness design for long-running application development.
• LangChain, Improving Deep Agents with harness engineering.
• Yang et al., SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering.
• Wang et al., Harness Handbook: Making Evolving Agent Harnesses Readable, Navigable, and Editable, arXiv:2607.13285, 2026.
• AWS, Making retries safe with idempotent APIs.
• Model Context Protocol, Tools specification.
• Market Analyst Agent Repository

The Market Analyst Agent code is on GitHub.