A benchmark built from real coding sessions just measured the gap between passing checks and finishing the job. The best coding agent in the Co-Design Bench study passed 69.9 percent of criteria on average. It fully finished only 26.4 percent of 273 real tasks. The gap is 43.5 points. Grading used 2,331 hidden expert criteria, judge then refute, in a live browser. The paper is “Co-Design Bench: A Benchmark for Evaluating AI Agents on Real-World Collaborative Coding Tasks” by Wei et al., submitted October 26, 2026.
The 46-second reel: hype says seventy percent, receipts say twenty-six.
69.9 to 26.4
The headline number is the gap, not either side of it. The tasks came from 26,314 real coding sessions, deduplicated to 37,463 queries, then curated down to 273 tasks: 100 multimodal editing tasks and 173 repository tasks. Each task was graded against hidden expert-written criteria, an average of about eight or nine per task, with a judge then refute step in a live browser. The top model, GPT-6 Astra, passed 69.9 percent of those criteria on average. It fully finished 26.4 percent of the tasks. An agent that passes most checks and finishes a quarter of the work is not a 70 percent agent. It is a 26 percent agent with a generous average.
Complexity made it worse
The repository tasks show the shape of the failure. With one or two edit types, strict completion was 65.4 percent and tasks where nothing passed were 0.9 percent. With three or four edit types, strict completion fell to 59.1 percent and zero-pass tasks rose to 5.8 percent. Zero-pass tasks jumped 6.4 times as the edit scope widened. That is observational, not causal: the paper’s own caveat is that these comparisons are not controlled effects. But the direction is the same one every data engineer already knows from production. A pull request that touches one thing fails in one way. A pull request that touches four things fails in four ways at once.
The caveats are part of the number
The study is honest about what it does not prove. The hard cases were deliberately picked, so the tasks skew harder than a random day of coding. Each task had one fixed request with no follow-ups, which is not how real collaboration works. AI judges can err, even with the refute step. And the authors are Alibaba-led while the top model was not Alibaba’s, which is worth knowing and does not change the measured numbers. None of this rescues the average. It just keeps the 26.4 percent from being read as a law of nature. It is a measurement of a hard benchmark, graded strictly, in a real browser.
The fix is a gate, not a better average
The fix we use follows directly from the measurement. Gate on full-pass rate, not the average. An agent PR that passes 70 percent of checks and breaks the flow in a browser test is a failed PR. Scope pull requests to one or two edit types, where zero-pass stays under one percent. That is our recommendation, not the paper’s. The paper measures. The gate is the production answer to what it measured.
The test: would you merge it?
The question this study leaves you with is a merge decision, not a model ranking. Would you merge an agent PR that passed 70 percent of checks if a browser test caught a broken flow? If the answer is no, then the number you should be tracking was never the average. It was the full-pass rate. Count how many of your agent’s PRs finish completely, end to end, in a real environment. If you do not know that number, the 69.9 percent is doing exactly what averages do: hiding the work that is not done.
This post started as an Instagram post →
- https://www.instagram.com/reel/Dfv43iQCbOw/
- https://www.threads.com/@dataeng.ai/post/Dfv5ABcpXil
- https://www.youtube.com/shorts/wz40Fts0m-g
- https://arxiv.org/abs/2610.35138
- https://codesignbench.github.io/
Numbers above trace to these sources. If one moved, tell us and we fix it.


Talk it through
Argue with us on Instagram.