The harness changed the bill, not the grade
HarnessTax (UC Berkeley and Arena, 2026-09-16) ran seven models through Claude Code, Codex CLI and Pi on SWE-bench Lite and Terminal-Bench 2.0 samples and found harness choice had little effect on success holding the model fixed, while cost moved up to fivefold. The same window Claude Code stopped offering its task-tracking tools on current models (2.1.268), removed TaskOutput and read AGENTS.md (2.1.277), and Pi's rewritten harness reached a tag in v0.85.0.
What this changes for operators
- For API-metered work, measure your own tasks in a minimal harness before assuming the vendor harness earns its overhead. The study is 30 tasks per benchmark; long-horizon and subscription use are outside it.
- Hooks or transcript parsers keyed on TodoWrite or TaskOutput lose their signal on current Claude models; delete CLAUDE.md shims that only point at AGENTS.md once past 2.1.277.
Primary sources
- third_party_research 2026-09-21-pi-coding-agent-harnesstax-study-same-model-same-success-up-to-five-times-the-cost arena.ai/blog/coding-agents-harness-tax
- third_party_research 2026-09-21-pi-coding-agent-harnesstax-study-same-model-same-success-up-to-five-times-the-cost harnesstax.github.io/
- github_release 2026-09-21-claude-code-agents-md-support-and-task-scaffolding-removed-for-current-models anthropics/claude-code / v2.1.277
- github_release 2026-09-21-claude-code-agents-md-support-and-task-scaffolding-removed-for-current-models anthropics/claude-code / v2.1.268
- tagged_commit_file 2026-09-21-pi-coding-agent-the-dev-harness-reached-main-and-a-tag-in-v0-85-0-its-plugin-surface-was-published-to-npm earendil-works/pi / packages/agent/docs/plugins.md
- github_release 2026-09-21-pi-coding-agent-the-dev-harness-reached-main-and-a-tag-in-v0-85-0-its-plugin-surface-was-published-to-npm earendil-works/pi / v0.85.1
Signal metadata
Source findings
- 2026-09-21-pi-coding-agent-harnesstax-study-same-model-same-success-up-to-five-times-the-cost 2026-09-21-pi-coding-agent-harnesstax-study-same-model-same-success-up-to-five-times-the-cost
- 2026-09-21-claude-code-agents-md-support-and-task-scaffolding-removed-for-current-models 2026-09-21-claude-code-agents-md-support-and-task-scaffolding-removed-for-current-models
- 2026-09-21-pi-coding-agent-the-dev-harness-reached-main-and-a-tag-in-v0-85-0-its-plugin-surface-was-published-to-npm 2026-09-21-pi-coding-agent-the-dev-harness-reached-main-and-a-tag-in-v0-85-0-its-plugin-surface-was-published-to-npm
Featured in
- Before the First Turn / 2026-09-21
Run: 2026-09-21-weekly-digest-2026-08-20_2026-09-21-frontier-v0
Schema: bitter.frontier_signals.v0 / ID: 2026-09-21-the-harness-changes-the-bill-not-the-grade
Research evidence and publication history are open in the repository.