2026-07-22 · Updated 2026-07-22

Harness engineering for AI agents: designing the loop around the model

Part 6 of the Engineering the Agentic Stack series

An agent’s reasoning loop chooses the next action. Its harness supplies context, validates and authorizes proposed tool calls, dispatches accepted calls to the runtime, records the results, and decides whether the task is complete.

Models propose edits and may declare tasks complete, while the harness enforces permissions and interprets the acceptance checks that decide whether the loop can stop. Part 5 covered the runtime that keeps the process alive. This article focuses on the control code inside that runtime: how to measure whether each addition helps and how to keep the resulting controls locatable as the harness changes.

Beyond the first explicit acceptance check, every added retry, handoff, or evaluator is a hypothesis about an observed failure. It earns a place only when a controlled comparison shows that it helps.

To make that control path concrete, I’ll use a small fictional store repository. The coding task is to lower the threshold for a 10% automatic discount from $100 to $75 in src/checkout.py. The repository has two required checks:

• pytest tests/test_checkout.py verifies the discount calculation.
• pnpm playwright test tests/checkout_discount.spec.ts adds an $80 item in a local test store and checks that the checkout page shows an $8 discount.

The example is a teaching fixture, not a real application or benchmark. Every attempt starts from the same commit and seeded test data. The harness may accept the change only when both commands pass and the trace ties those results to the tested commit.

Later sections deliberately switch to two other examples. A staging-order API shows why state-changing calls need replay protection, while a payment-adapter migration shows what a new model session needs to resume unfinished work. The companion lab near the end is separate again: it is runnable, but it contains generic simulated tasks rather than this store repository.

TL;DR: Start with one model, a few narrow tools, and an explicit acceptance check. Add retries after traces show transient call failures. Add a progress handoff when resumed sessions repeat work. When you test either change, keep the model version, tasks, grader, and total budget fixed. Remove the component when it does not improve the measured result.

The diagram follows the discount change from proposal to evidence. The harness supplies the task and files, checks the proposed edit_file arguments and permissions, and dispatches the accepted call. After the runtime applies the edit, the harness runs the named unit and browser acceptance tests. A failed command goes back to the model as evidence for another turn; two passing commands make the change eligible for acceptance.

What the harness owns

OpenAI’s Codex loop walkthrough describes the basic cycle. The harness assembles a prompt, asks the model for the next action, sends an accepted tool call to the runtime, appends the result, and asks again until the harness accepts the result or hands control back to the user.

Implementations may merge several responsibilities into one process. The failure boundaries are still different:

When a failure appears, diagnose the boundary that should respond. A poor plan may need better instructions or model reasoning. If edit_file targets a path outside src/, the harness should reject it, while a sandbox process that dies before the edit runs belongs to the runtime, which must restart the worker or report the crash.

OpenAI’s own harness-engineering case study describes a bootable application instance for each worktree. The team also wired browser automation into the agent environment and exposed logs, metrics, and traces.

A task such as “no span in these four critical user journeys exceeds two seconds” became testable because the agent could run the application and query the same signals an engineer would inspect. The case study is product-specific. What transfers is the condition behind the result: the application and its performance signals had to be available inside the agent environment.

Lopopolo, the author of that case study, maintains a field guide for harness engineering that names the two levers this article pulls: hold the model and coding agent fixed as a black box, and engineer the context and tools around them. His framing also explains why so much of the harness ends up as ordinary code.

An organization’s quality bar, procedures, exception history, and authority relationships sit below the waterline of what a general model can know. The harness surfaces them as repository instructions, permission rules, and acceptance checks. Each accepted run can feed its lessons back into those artifacts instead of relying on the next session to rediscover them.

Follow the discount change from proposal to acceptance

For the discount task defined above, the model proposes changing calculate_discount in src/checkout.py. Several things happen before that edit counts as progress:

1. The context builder supplies the task, repository instructions, relevant files, prior tool results, and the current plan.
2. The model proposes an edit_file call with a path and replacement text.
3. The tool boundary (the harness code between proposal and execution) validates the arguments, checks the path against the allowed scope, and asks for approval if the operation needs it.
4. The runtime applies the edit in the sandbox and returns a structured result.
5. The harness runs pytest tests/test_checkout.py, followed by pnpm playwright test tests/checkout_discount.spec.ts, and reads both exit codes. The browser test checks the visible $8 discount on the seeded $80 cart.
6. The harness decides what the results mean. A failed check becomes new context for the next model turn, and a passing run makes the task a completion candidate.
7. A passing result becomes completion evidence only after the harness records the command, exit code, and tested artifact version in the trace.

At step 2, no file has changed. The harness can reject ../../secrets.env, require approval for a destructive command, or stop a run that has exhausted its budget. After the tests run, the harness reads their exit codes itself. The model cannot mark its own edit as passing.

The trace should show the proposed path and replacement text, the permission decision, the files that changed, the tested commit, and both command results. A final done message without those records does not prove that this change passed its required checks.

Decide where each rule is enforced

Put tests/checkout_discount.spec.ts in normal program logic. The harness dispatches the Playwright command to the runtime, reads its exit code, and refuses to end the run while it fails. A prompt can remind the model to run the test. It cannot stop the model from declaring success without evidence.

Other rules fit different layers:

Tool contracts separate proposal from permission

The discount task only needs file edits and test commands. A state-changing API has a different failure mode, so switch examples for this section. Suppose the agent can call create_test_order against a staging order service while setting up test data. This tool is not one of the discount task’s acceptance checks. It is useful here because a timeout can hide whether the service created an order.

The tool boundary needs more than a natural-language description. For create_test_order, the harness needs a contract with:

• validated arguments, so malformed input is rejected before execution
• a structured result such as { "order_id": "123", "created": true }, so later checks do not have to parse free-form text
• an effect category that records whether the call only fetches information or changes a file, database record, or external service. It also records whether repeating the call is safe. This label tells the harness whether an automatic retry could duplicate work: it can retry get_order_status when the service defines that lookup as read-only, but it must not blindly retry create_test_order because the first call may already have created the order
• a timeout and retry policy, so a lost response does not trigger an unlimited sequence of calls
• a permission rule that says what approval is required. Reading order status may run automatically, while creating an order may require confirmation

The natural-language description is text shown to the model. It might say, “Create a test order for checkout verification.” That sentence helps the model decide when to propose create_test_order. It does not authorize the call. In this example, the harness’s MCP client validates the arguments, applies its own rules, and checks server trust, approval requirements, and retry safety before dispatching anything.

An MCP server publishes tool descriptions and optional behavior annotations to the client. A faulty or malicious server could describe a state-changing tool as harmless. If the client accepted that claim automatically, it might run or retry create_test_order without approval and create a duplicate, so the MCP specification requires clients to treat tool annotations as untrusted unless the server itself is trusted.

The specification does not prescribe one universal trust setting. The client therefore needs an explicit trust policy for its deployment; a server cannot make its own annotations trustworthy. That policy decides which metadata may influence permission or retry decisions and which annotations remain advisory only.

Retrying a state-changing call needs replay protection

A harder retry problem appears when create_test_order creates the order but its HTTP response is lost. The harness sees a timeout and cannot tell whether the server completed the request. Repeating the call may create a second order.

A status lookup can be retried when the service defines it as read-only. A creation call needs protection such as an idempotency key: the client attaches a unique request identifier, and the service returns the first result instead of creating another order when it sees that identifier again. Without this protection, the harness should check whether the order exists or ask for a human decision before another attempt. AWS documents this pattern in its idempotent API guidance.

Acceptance needs independent evidence

A successful create_test_order response establishes only that the tool returned data. It does not prove that a coding task passed its tests. If a later browser test depends on the staged order, the harness must validate the response schema and still run that test before accepting the code change.

Some criteria cannot be reduced to an exit code. For a separate visual-design task, a fresh evaluator can compare a rendered page or diagram with a written rubric. Teams should check that evaluator against human reviews before using it as a completion gate.

A payment-adapter migration needs a handoff

Switch tasks again, but stay in the fictional store repository. The agent now has to migrate checkout from payment adapter v1 to v2. The work spans the checkout handler, payment client, configuration, and tests, so it may outlast one model session.

Before the first session reaches its context limit, it has modified several files, started a local payment sandbox, and left tests/payment_migration.spec.ts failing. That browser acceptance test completes one payment through adapter v2 and verifies the recorded provider ID. A conversation summary can orient the next model session, but it cannot restart the sandbox or prove which files are currently modified.

The next session has to recover three things:

Compaction replaces older messages with a shorter summary so the current session can continue. A progress handoff records what the next session needs: the current branch, changed files, last test command and output, and the next unresolved step. If the old conversation contains stale assumptions, the harness can start a fresh model session with that handoff and the current workspace. Replacing a crashed worker and restoring its processes is a separate runtime recovery job.

A small documentation edit may need none of these mechanisms. The payment migration needs a handoff once work crosses sessions because the next model session must reconstruct both the workspace and the task status.

Anthropic’s experiments with long-running coding agents used git history and a progress file between sessions, while its later harness-design report separates compaction from a fresh-context handoff and reports the additional orchestration, token use, and wall time required by handoffs.

Use traces to distinguish three failures

The next three rows are illustrative trace sketches, not measured runs or output from the companion lab. Each row shows a different failure and therefore a different harness response.

This distinction matters because a 503 does not make every call safe to retry. The first row is a read-only lookup. The second row is a state-changing request, so the idempotency key and server-side status decide whether another creation attempt is allowed. The third row is not a tool failure at all; the harness has not yet collected the evidence required to accept the discount change.

A chat transcript records what the model saw. It cannot prove whether the order service committed a request before the response disappeared. The trace must join the client call, approval decision, idempotency key, server result or status lookup, tested commit, and acceptance-test result. Those fields tell the harness which of the three paths it is on.

Before adding a component, name the repeated failure it should reduce and the number you will track. Remove the component if a controlled comparison does not move that number enough to repay its cost.

Measure one change at a time

An ablation measures whether a harness component causes the expected effect by changing or removing that component while the rest of the experiment stays fixed. For example, does editor linting help this model on this task suite?

Use the following protocol:

1. Freeze the model version, task instances, environment, grader, and prompts outside the component under test.
2. Give both variants the same total token, time, and dollar budget.
3. Choose the number of trials or stopping rule before running the comparison.
4. Run the same task instances in both variants. Because model outputs vary, repeat each task several times.
5. Report the mean together with the spread or confidence interval.
6. Count every started trial, including timeouts, policy stops, harness crashes, and evaluator failures.

Success rate alone can hide an expensive component. At minimum, track broken tasks accepted as complete, cost and wall time per completed task, tool errors, duplicate orders, review minutes, and manual permission overrides. Choose the metric that carries the real cost for the product. A two-point rise in completed tasks is a bad trade if it doubles the review queue.

A paired payment-migration experiment makes the progress handoff measurable. Each control/treatment pair starts from the same repository commit and seeded checkpoint, with the same model, task, grader, and total budget. The handoff is the only switch. The primary metric counts duplicate tool actions after resume: an action is duplicate when its operation and artifact match a step that the previous session had already completed.

The SWE-agent paper fixes GPT-4 Turbo on the 300-task SWE-bench Lite split and reports 18.0% resolved with its full interface, compared with 11.0% for the shell-only agent. The paper also changed individual interface features:

These numbers belong to that model, benchmark, and $4 per-task cap. The without linting, full file, and full history rows are the useful one-feature tests: each changed an interface feature while the model and evaluation setup stayed fixed.

LangChain published a broader fixed-model comparison for deepagents-cli. It reports a Terminal-Bench 2.0 increase from 52.8% to 66.5% with gpt-5.2-codex fixed while its team changed the system prompt, tools, and middleware. The post bundles several changes and omits a confidence interval, fixed-total-budget comparison, and per-change ablation table. That result cannot identify the useful change.

Anthropic’s long-running application report is a qualitative, product-specific case study rather than a controlled benchmark. Its evaluator checked 27 level-editor criteria in Sprint 3. The team reports that evaluator calls became overhead on tasks Opus 4.6 could complete reliably alone but still helped near the model’s edge, so it removed harness components one at a time after the model upgrade. The example is a reason to revalidate old scaffolding when the model changes; it does not estimate a general effect size.

Keep the harness editable after it earns its place

Ablation keeps a harness small, but its code can still outlive the model it was tuned for. A request such as “mask secrets in every capture path” names a behavior, not a file. In a production harness, that behavior may span execution stages and shared state. A human or coding agent must find every implementation site before changing it safely.

A 2026 preprint from Wang et al., the Harness Handbook, calls this step behavior localization. The handbook builds a behavior-centric map from the harness codebase. Static analysis extracts a program graph without model calls, then an LLM organizes its units into execution stages.

The maintainer or coding agent starts with a system overview, opens the relevant execution stage, and descends to source-grounded entries for a function or file. A register view records where shared state is written and read between stages. This hierarchy keeps the overview small while preserving a path to source.

Freshness is a separate rule. Every locator must resolve against the live repository. The handbook excludes stale entries instead of guessing, and each non-empty diff resynchronizes the entries it affects.

The diagram compresses the modification loop: a behavior-only request descends the handbook’s levels, every candidate locator is verified against the live repository before the plan is written, and each applied diff resynchronizes the map.

The Handbook evaluation follows the protocol this article argues for. On two open-source harnesses (Terminus-2, six Python files, and the Codex monorepo, 2,267 Rust files), a read-only planner powered by DeepSeek-V4-Pro either explored the repository directly or routed through the handbook. Requests, repository, tool permissions, and decoding were identical in both arms. Three judges (GPT-5.5, Opus 4.8, DeepSeek-V4-Pro) scored each edit plan on localization, scope control, and reasoning:

The handbook-assisted planner won more often and used fewer planner tokens in both repositories. The conditions stay attached to that result: three LLM judges scored edit plans produced by one planner model on two harnesses. The study evaluated plans, not executed diffs or production defect rates.

The earlier sections use traces to answer why a component exists. This map answers the next question: where does that component live when it has to change?

Try the method in the companion lab

The harness-demo project at commit 517353f3 is a small, deterministic exercise with 12 generic synthetic tasks covering code changes such as fix-parser-edge-case, split-large-module, and wire-browser-test. It does not implement the fictional store repository.

Each task fixture declares a difficulty plus four boolean conditions: a flaky tool, lost progress, a missed implementation gap, and ambiguous completion. The simulator derives a fifth condition for difficult tasks that also need a progress file: without context_reset, compaction preserves stale assumptions. A deterministic grader marks a task as passed only when the selected configuration handles every applicable condition. No model or external service runs.

The commands answer different questions:

• make check runs Ruff and seven unit tests, including the validator that rejects any ablation pair which changes more than one component.
• make run prints a cumulative teaching matrix, then five valid leave-one-component-out comparisons.
• make failures names the unhandled condition for every failed task. The full harness should end with all synthetic tasks pass.

make check
make run
make failures

The causal section of make run looks like this:

component                 control  treatment  delta
retry_policy              8/12     12/12       +4
progress_handoff          7/12     12/12       +5
evaluator                 8/12     12/12       +4
fail_closed_acceptance    7/12     12/12       +5
context_reset            10/12     12/12       +2

For each row, the control is the full configuration with one component removed; the treatment restores only that component. The earlier cumulative matrix is useful for orientation, but some of its adjacent rows add several components at once and therefore cannot identify a cause.

The lab validates every declared pair before running it. Its regression tests also include an intentionally invalid pair that changes retry policy and evaluator together; the validator rejects it.

The lab’s configuration schema checks all five component fields. This runnable excerpt shows the same guard on one valid progress-handoff pair:

from dataclasses import dataclass, fields


@dataclass(frozen=True)
class Config:
    progress_handoff: bool = False
    evaluator: bool = False
    retry_policy: bool = False
    fail_closed_acceptance: bool = False
    context_reset: bool = False


def changed_components(control: Config, treatment: Config) -> tuple[str, ...]:
    return tuple(
        field.name
        for field in fields(control)
        if getattr(control, field.name) != getattr(treatment, field.name)
    )


control = Config(progress_handoff=False, evaluator=True, retry_policy=True)
treatment = Config(progress_handoff=True, evaluator=True, retry_policy=True)
assert changed_components(control, treatment) == ("progress_handoff",)

Start with one loop and one acceptance check

I would start a coding-agent harness with one capable model, repository instructions, a few narrow tools, a sandbox, and one explicit acceptance test. I would record tool calls, results, costs, and that final test in one trace so the first useful failures are visible without reconstructing them from terminal logs and chat transcripts. This is a proposed baseline, not evidence from a deployed system.

From there, add only what a trace justifies. Record who maintains each component, how many tokens or seconds it adds, and which regression test would justify removing it after a model upgrade.

Six months later, someone who sees progress_handoff=True should be able to find the failed traces that justified it and the regression cases that still keep it there. The traces explain why the component exists; a current behavior map explains where to touch it.

References

• OpenAI, Unrolling the Codex agent loop.
• OpenAI, Harness engineering: leveraging Codex in an agent-first world.
• Lopopolo, Harness engineering: anthology, field guide, and agent context bundle.
• Anthropic Engineering, Effective harnesses for long-running agents.
• Anthropic Engineering, Harness design for long-running application development.
• LangChain, Improving Deep Agents with harness engineering.
• Yang et al., SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering.
• Wang et al., Harness Handbook: Making Evolving Agent Harnesses Readable, Navigable, and Editable, arXiv:2607.13285, 2026.
• AWS, Making retries safe with idempotent APIs.
• Model Context Protocol, Tools specification.

Series: Engineering the Agentic Stack

• Part 1: AI Agent Reasoning Loops in 2026: ReAct, ReWOO, and Plan-and-Execute
• Part 2: AI Agent Memory Architecture in 2026: checkpoints, vector stores, and document memory
• Part 3: AI Agent Tool Use in 2026: MCP, CLI, Skills, code execution, and ACI
• Part 4: AI Agent Security in 2026: guardrails, permissions, sandboxes, HITL, and MCP scoping
• Part 5: Long-Running AI Agent Runtime in 2026: sessions, sandboxes, checkpoints, harnesses, and deployment shapes
• Part 6: Harness Engineering for AI Agents (this post)