The Agent Harness Is Getting Out of Control

I have been building AI agents for a while now. The first version always feels simple. Give the model some instructions, define a few tools, and keep calling the model until it returns an answer instead of another tool call.

user request
    -> model
    -> tool call
    -> tool result
    -> model
    -> final answer

That is an agent, at least in the smallest useful sense. Then the agent has to survive contact with a real application.

What happens if a tool fails? How many times should the loop continue? Where does the state live if the process restarts? Can the model call the refund tool without approval? How do we know why it called the tool? What if the conversation no longer fits in the context window?

Each question adds another piece around the model. Eventually, the model API call is the easiest part of the system.

The Idea

The software around the model is often called the agent harness. It connects the model to tools and the outside world. Depending on the application, it may also manage:

Model APIs and streaming
Tool definitions, execution, and errors
The agent loop and its stopping conditions
Conversation and task state
Context selection and compaction
Short-term and long-term memory
Prompts and model-specific instructions
Permissions and human approvals
Guardrails and safety policies
Retries, budgets, tracing, and evaluation

This is a strange amount of responsibility for something that started as a while loop.

It does not matter much whether we are building a coding agent or a customer support agent. The same goes for an assistant over a company wiki. The tools and policies change, but the harness has the same job. It has to turn a stateless model call into a stateful system that can act over time.

The model generates the next action. The harness has to make that action part of a reliable system.

I used to think of the harness as glue code. Glue code connects stable components. An agent harness is closer to a small runtime whose most important dependency changes behavior every few months.

How the Harness Grows

Suppose we are building a customer support agent. The first version has a system prompt, a search tool, and a refund tool.

The model searches forever, so we add a turn limit. It calls tools with invalid arguments, so we validate. The API times out and the model retries, so we add idempotency. Old messages pile up, so we add context compaction. It forgets prior cases, so we add memory. It refunds too eagerly, so we add an approval gate. Nobody can explain a bad decision, so we add tracing.

None of these additions are unreasonable. That is the problem. The harness grows through a series of sensible local decisions.

The same thing happens with a coding agent. Replace the refund API with a shell, the customer history with a repository, and the approval gate with file permissions. The tools change. The machinery underneath does not.

Agent frameworks package these pieces, which helps at the start. It also makes it easy to turn on routing, reflection, planning, and memory before measuring whether any of it helps. Anthropic's advice is surprisingly conservative: start simple, add complexity when evaluation shows that it helps. That is hard to follow when every harness feature sounds like something a serious system should have.

Two Kinds of Complexity

Some of this complexity belongs in the application and will not go away.

A better model does not remove the need for permissions. It should not decide whether it is authorized to refund $2,000, delete a production database, or read a private document. Durable state, audit logs, idempotency, cost limits, and human approval are ordinary production concerns. The model may suggest an action, but the application still owns the consequences.

Other parts of the harness exist because the current model cannot reliably do something itself.

We ask the harness to split a task into steps, force the model to reflect, route work to specialist agents, summarize its context, choose which memories to retrieve, retry a malformed tool call, or follow a carefully arranged reasoning template. These techniques can improve performance, but they are tied to the behavior of a particular generation of models.

That gives us two rough categories:

Product infrastructure:
    permissions, durable state, audit, verification, approvals

Model scaffolding:
    reasoning prompts, planners, reflection loops, routers, context tricks

The first category should be stable. The second should be easy to replace or delete. Some pieces sit between both. Context compaction is a product concern because cost and latency demand it, but how you compact depends on the model. The boundary is not perfectly clean. I think even a rough version of it is better than treating the entire harness as one blob. In practice, the two categories often end up mixed inside one framework, one runner, and one large system prompt.

Chain of Thought Crossed the Boundary

Chain-of-thought prompting is a useful example of how quickly this boundary moves.

In 2022, prompting models to "think step by step" improved reasoning. ReAct extended this into an agent loop:

thought -> action -> observation -> thought -> action

The harness parsed each action, ran the tool, and asked the model to reason again. For a while, this was almost synonymous with building an agent.

Then reasoning models arrived. OpenAI's o1 and DeepSeek-R1 trained chains of thought into the model itself through reinforcement learning. Something we used to construct outside the model became a native capability.

Google's Gemini migration guide now recommends replacing complex chain-of-thought prompts with simpler instructions and native thinking controls.

The old prompt became worse than useless. It could hurt the new model.

Upgrading the Model Is Not a Dependency Bump

This is what makes harness engineering feel unstable. A new model is not a faster implementation of the same interface.

One model wants explicit planning instructions. Another already plans internally and overthinks when given the same prompt. We can standardize the JSON shape of a tool call, but we cannot standardize what makes a model use tools well.

Harness-Bench and other recent benchmarks support this. Performance changes substantially across model and harness pairings. A strict scaffold can help one model and hurt another. So when we say a model scored some percentage on an agent benchmark, part of that score belongs to the harness.

Building for Deletion

I do not think the answer is to wait for models to absorb the whole harness. Models should not own permissions or transactions, and they should not be the authoritative state of an application.

But the boundary should be obvious in the code. The durable part owns events, tool execution, policy, and verification. A thinner model adapter owns the prompts and context formatting, plus whatever else is specific to one model. An evaluation suite tests the pairing before an upgrade reaches production.

Every piece of scaffolding should have a reason to exist. If a reflection step does not improve the evaluation, remove it. If the model can choose tools without a router, remove the router. If native reasoning replaces chain-of-thought instructions, delete them.

This is harder than adding features. Old scaffolding feels safe because it once fixed a real failure. Without evaluation, nobody knows whether it still helps.

At first, I thought a good agent harness was one that handled every possible model failure. Now I think that is how the harness gets out of control. It remembers every weakness of every model we have used, long after the model is gone.

The harness needs memory too, but unlike the agent, it should be much better at forgetting.