Complex AI coding harnesses offer little accuracy gain, up to 3x higher cost
A study of 492 SWE-bench Verified tasks found feature-rich coding agent harnesses did not consistently outperform lightweight ones, with per-task costs varying threefold.
Researchers from China's Shandong University and UC Berkeley have published a study comparing three coding agent harnesses — Anthropic's Claude Code, open-source OpenCode, and the minimal mini-SWE-agent — and found that feature-rich harnesses did not consistently score higher than lightweight ones, while per-task cost could be up to three times higher, according to aitimes.com.
The study, posted on an online archive, evaluated 492 offline SWE-bench Verified tasks, splitting out 45 hard tasks and the remaining 447. Models tested included open-weight Qwen3.6-35B-A3B and Qwen3.8-27B, API-based DeepSeek-V4-Flash, GLM-5.3-Flash and HY4-Preview, plus Claude Opus 5 run in Claude Code as a reference.
Roughly 13% of hard-task results flipped between success and failure both when the harness changed and when the same harness was re-run. OpenCode scored up to 9 percentage points lower than other environments on large tasks, about half of that gap caused by hitting the 32,000-token output limit before writing a patch, while Claude Code had a recovery feature that automatically asked the model to continue output.
With the same model and task, per-task cost varied up to threefold depending on the harness, a gap present from the first API call. Fixed context in system prompts and tool definitions exceeded 10,000 tokens for Claude Code versus roughly 800 tokens for mini-SWE-agent. DeepSeek's official command-line harness stayed within the success range of other harnesses but had the highest per-task cost among those compared, and with DeepSeek models, mini-SWE-agent became more expensive than other harnesses because of more calls.
The researchers recommended repeated testing with enough tasks and simplifying system prompts and tool definitions rather than adopting complex harnesses by default.
Quick answers
What benchmarks were used in the study?
The study evaluated 492 offline SWE-bench Verified tasks, including 45 hard tasks and 447 remaining tasks.
Which models were tested?
Models tested included Qwen3.6-35B-A3B, Qwen3.8-27B, DeepSeek-V4-Flash, GLM-5.3-Flash, HY4-Preview, and Claude Opus 5 as a reference.
What did the researchers recommend?
They recommended repeated testing with enough tasks and simplifying system prompts and tool definitions rather than adopting complex harnesses by default.