HarnessTax: How Much Does the Harness Matter for Coding Agents?

❓ The Question: What Does the Harness Add?

Language models are changing how we build software, and coding agents put those capabilities to work through a harness — the software system that manages a model's tools, context, and task execution. While models provide the core intelligence, harnesses are increasingly seen as central to how effectively that intelligence is used. Choosing a coding agent means selecting both a model and a harness — even when you think you're only choosing a model.

Millions of people already use coding agents, yet the impact of harness choice has remained unclear. The question the paper sets out to answer is direct: could another harness help the same model solve more tasks, or reduce costs?

🧪 Experiment Setup: 21 Pairs, Two Benchmarks

The study compares 21 model–harness pairs — seven models across three harnesses — on two open-source benchmarks:

The seven models span three families: OpenAI's GPT-5.6 Luna, GPT-5.6 Sol, and GPT-5-Codex; Anthropic's Claude Fable 5, Sonnet 4.6, and Opus; plus the open-weight Kimi K3 (accessed via Fireworks AI, in its single native thinking mode).

Methodology, in brief: 30 randomly sampled tasks per benchmark, three runs per task to capture variance, each harness's native configuration at its high-effort setting, capped at 100 agent turns per attempt. Success is measured with each benchmark's official evaluator. Token costs use a fixed direct-API price list dated September 1, 2026 — the same prices applied to each model across harnesses, so cost differences trace to the harness, not to pricing. Confidence intervals come from 10,000 bootstrap resamples.

💰 Finding 1/3 — Harness Moves Cost More Than Correctness

The headline: the same model often achieves a similar success rate at substantially different costs. Claude Fable 5 solves 97.8% of attempts in Claude Code, 96.7% in Codex, and 96.7% in Pi — yet Claude Code costs about twice as much as Pi ($1.33 vs $0.67). A 1.1-point success-rate edge for double the money.

On absolute quality, GPT-5.6 Luna offers the lowest cost on both benchmarks, while Claude Fable 5 reaches the highest success rate on SWE-bench Lite. Kimi K3, the open-weight model, sits close to the Pareto frontier near GPT-5.6 Sol on SWE-bench Lite and just below it on Terminal-Bench 2.0 — a reminder that open weights are now genuinely competitive in this arena.

🛠️ Finding 2/3 — A Simple Harness Can Be Competitive

The second finding is the most striking for anyone building on a budget: Pi reaches the Pareto frontier on both benchmarks by providing just four tools — read, write, edit, and bash. No elaborate scaffolding, no proprietary magic.

Where does the cost actually go? The paper decomposes spending across completed attempts, recorded turn counts, and initial context. The key observation: agents can take similar numbers of turns at substantially different costs. For Fable 5 on SWE-bench Lite, Pi and Claude Code average 15.4 vs 15.3 turns per attempt — essentially identical — yet Claude Code costs about twice as much. That points to higher spending per turn rather than more turns.

The practical implication for research: the effectiveness of Pi and Codex shows that open-source harness research with existing models is a real, accessible frontier. Researchers can work with state-of-the-art coding harnesses without access to proprietary harnesses or co-training with the model. Richer harness features may still pay off for other models or workloads — but complexity should be treated as an empirical trade-off, not an assumption.

🔀 Finding 3/3 — Models Win Outside Their Own Harness

The third finding overturns a common assumption: provider-specific optimization does not guarantee the best pairing. Providers sometimes optimize models for their own coding environments — OpenAI, for example, describes GPT-5-Codex as optimized for software engineering in Codex. Yet across the six Anthropic and OpenAI models and both benchmarks, an alternative harness achieves the highest observed success rate in nine of twelve comparisons.

The takeaway: a model's capabilities are compatible, generalizable, and can carry over to other harnesses. Sharing a provider with a harness does not guarantee the best pairing. The real question is empirical — which harness delivers the best balance of cost and task success for your model and workload.

🧾 The "Harness Tax" & What It Means for Evals

The paper borrows its central metaphor — the harness tax — from a Portkey piece on "dead weight inside your coding agent." The point: a hidden tax you pay when you accept a default harness without comparing. It goes unnoticed when you focus only on task success, because success barely moves; it shows up in your token bill.

This has a concrete consequence for how models are evaluated: model evaluations should compare the same model's cost and task success across commonly used harnesses, not just report a single harness's numbers. A model that looks expensive on one harness can be competitive on another — at half the cost.

🔭 What This Means Going Forward

The authors are careful about scope: the findings are limited to the two open-source benchmarks tested, which the models may have encountered during training — results may differ on other benchmarks and workloads. And harness selection matters even more in real development workflows, where requirements evolve, developers give feedback, and tasks stretch across sessions.

The broader framing is the most interesting part. For day-to-day tasks, coding agents are essentially interfaces to model intelligence — they manage context, access tools, and execute tasks. As models get more capable, they may need less of today's scaffolding. General-purpose coding agents should therefore prioritize cost efficiency and reliability, because many tasks don't need fancy add-on features.

For harder problems at the boundary of a model's capabilities — including scientific discovery — agents may still benefit from harnesses that provide structured guidance for exploring ideas, evaluating candidates, and learning from feedback. Harness research, in that light, is a way to help models push the boundaries of knowledge.

💡 Key Takeaways

1. Harness barely moves success, but can swing cost up to 5×. The same model reaches near-identical success rates across harnesses — at very different prices.
2. "Your Claude models may not need Claude Code." Claude Code cost ~2× Pi and ~1.6× Codex on SWE-bench Lite for a ~1% success edge.
3. A minimal harness is competitive. Pi — four tools: read, write, edit, bash — reaches the Pareto frontier on both benchmarks.
4. The tax starts at the first call. Claude Code's mean initial context is over 10× Pi's, inflating cost from turn one.
5. Models win outside their own harness. In 9 of 12 comparisons, an alternative harness beat the provider's own pairing.
6. Open weights are competitive. Kimi K3 sits near the Pareto frontier alongside GPT-5.6 Sol.
7. Evals must compare harnesses. A model's cost/success should be reported across harnesses, not from a single default.
8. The future is an adaptive harness. Users shouldn't pick configurations; harnesses should adapt as tasks unfold.

📚 Citation & References

Cite the project as:

@misc{pan2026harnesstax,
  title  = {{HarnessTax: How Much Does Harness Matter for Coding Agents?}},
  author = {Pan, Melissa Z. and Yang, Shuo and Arabzadeh, Negar and Chiang, Wei-Lin and Stoica, Ion and Zaharia, Matei},
  year   = {2026},
  url    = {https://harnesstax.github.io/},
}

• 📄 harnesstax.github.io — the original article
• 🧪 SWE-bench Lite — swebench.com (Carlos E. Jimenez, John Yang, Jiayi Geng)
• 🖥️ Terminal-Bench 2.0 — Merrill et al., arXiv:2601.11868 (Jan 2026)
• 🤖 SWE-agent — John Yang et al., "Agent-Computer Interfaces Enable Automated Software Engineering," NeurIPS 2024
• 💰 "The Harness Tax: The Dead Weight Inside Your Coding Agent" — Siddharth Sambharia, Portkey (Apr 2026)
• 🧩 Pi coding agent — GitHub README (accessed Sep 16, 2026)

Authors: Melissa Z. Pan, Shuo Yang, Negar Arabzadeh, Wei-Lin Chiang, Ion Stoica, Matei Zaharia — UC Berkeley (Sky Lab) & Arena. Funding acknowledged from the Amazon AI Fellowship, Arena, and the Laude Institute.