Best Coding Agent Fully Solves Only 26.4% of Real Tasks in New Co-Design Bench Test

The best AI coding agent fully solved only 26.4 percent of real tasks in a new test built from actual user requests, even though it met 69.9 percent of the requirements on average. That is the headline finding of Co-Design Bench, a benchmark released by researchers from Alibaba and several Asian universities, posted as a paper on 26 October 2026 and covered in the press on 8 November. The gap between the two numbers is the story: an agent can do most of a job and still leave it unfinished.

A coding agent is an AI system that reads a change request, edits code across files, runs the result and tries to fix its own errors, rather than only suggesting a snippet for a human to paste in. Co-Design Bench tests that full loop, using 273 real change requests drawn from 37,463 deduplicated user requests across 26,314 real coding sessions, graded against 2,331 hidden acceptance criteria written by experts. The coverage is one day old at the time of writing, and the paper itself is two weeks old.

How the benchmark works

Co-Design Bench starts where other benchmarks stop. Most coding tests use synthetic tasks or textbook problems, and the best models now score above 99 percent on some of them. The authors argue that those tests no longer tell us how much work still lands on humans, because a model that can run, check and fix its own code can game a narrow test. So they built a test from real life instead.

Each task is a request to change an existing web app, of the kind a user makes in a live coding session. Some tasks come with images, audio recordings, videos or documents as input, so the agent has to work with more than text. Each request is fixed, so the agent cannot ask follow-up questions or receive corrections in the middle of the task. That is a deliberate choice, and it is also a limitation, because real developers do iterate with an agent. The benchmark measures one-shot performance, not a conversation

Two developers working together at a desk with laptops showing code, a water bottle, a coffee cup, sticky notes and a small plant

Illustration: Two developers working together at a desk with laptops showing code, a water bottle, a coffee cup and sticy notes

The leaderboard

Ten models were tested as coding agents, and the main comparison ran every model through Claude Code as the harness, which is the agent framework that turns a model into a working coding tool. OpenAI's GPT-6 Astra led the field. It satisfied 69.9 percent of the hidden acceptance criteria on average, and it fully completed 26.4 percent of the 273 tasks. Anthropic's Claude Opus 5 took the next spot, meeting 66.1 percent of criteria and fully completing 18.3 percent of tasks. Alibaba's Qwen3.8-Max took the next spot after Claude, and xAI's Grok 4.6 came last, fully completing 6.6 percent of tasks.

The two tracks inside the benchmark tell different stories. On localised multimodal edits, which are changes to one part of an app, GPT-6 Astra fully solved 41.0 percent of tasks. On repository-level edits, which require changing code across many files, it fully solved only 17.9 percent. Multi-file work is where agents struggle most, and that is also where real software work usually lives. A change that touches one file is a demo. A change that touches twenty files, with tests and shared state, is a job.

Model	Criteria met on average	Tasks fully solved
GPT-6 Astra	69.9 percent	26.4 percent
Claude Opus 5	66.1 percent	18.3 percent
Qwen3.8-Max	Next after Claude Opus 5	Not separately reported in coverage
Grok 4.6	Not separately reported in coverage	6.6 percent, last place

Why the numbers are lower than the headlines

There are two honest reasons the scores look low, and both are built into the design. First, the tasks were deliberately weighted toward hard cases. The authors selected candidates partly on signals of past failure, such as a user rephrasing a request two or more times, repeated follow-ups, or an explicit rollback of a change. The 273 tasks are therefore a stress test, not a random sample of coding work, and 26.4 percent is a stress-test number, not the success rate on an average to-do item. A developer who asks an agent to rename a variable will get a better result than a developer who asks it to rebuild a checkout flow.

Close-up of programming code on a computer screen with a person's hand at the bottom

Illustration: Programming code on a computer screen, with a hand at the keyboard

The caveats the publisher must honour

This is an unreviewed preprint, not peer-reviewed work, and that status should travel with every citation of it. The paper was posted to arXiv on 26 October 2026, and it has not yet been through independent peer review. That does not make it wrong, but it means the results have not been checked by outside reviewers, and they should not be presented as settled science.

There is also a conflict of interest to disclose. All three corresponding authors, and a majority of the author list, work at Alibaba Group. That is a clear conflict, because Alibaba sells models that compete with the ones being tested. The result cuts against the conflict, because the top-ranked model was not an Alibaba model but OpenAI's GPT-6 Astra, and Alibaba's own Qwen3.8 models ranked third and fourth. Even so, readers deserve to know who wrote the test, and this article states it plainly.

What this means for developers

The practical lesson is about workflow, not about which model to buy. If the best agent fully finishes only one task in four on hard, real-world requests, then the right way to use an agent is as a fast first draft, not as a finished product. Let it do the bulk of the work, then review every change, run the tests, and fix the details it missed. That is slower than the marketing promises, but it is faster than writing everything by hand, and it is honest about where the technology stands.

Our earlier coverage of the model race gives useful context. Our comparison of Gemini Carbon vs Claude Opus 5.5 vs GPT-6 Astra shows how close the frontier models have become, and our report on the GPT-6 Astra benchmark scores shows the same model looking very different on different tests. The lesson of both articles is that a single number never tells the whole story, and Co-Design Bench is a reminder of that lesson from the other direction. A model can top a leaderboard and still need a human to finish the job.

Illustration: A modern glass high-rise office building above a plaza

What to watch next

Until then, the honest summary is narrow and useful. On 273 hard, real change requests, graded against 2,331 hidden expert criteria, the best of ten models fully solved 26.4 percent of tasks. It met 69.9 percent of the requirements on average. Repository work was far harder than single-file edits, and the harness around the model mattered almost as much as the model itself. The Kimi K3.1 Pro launch shows how fast the model field is moving, but Co-Design Bench shows that the gap between a good model and a finished job is still wide, and that gap is where human developers still earn their pay.

Key Takeaways

Co-Design Bench, posted on 26 October 2026 by researchers from Alibaba and several Asian universities, tested ten AI coding agents on 273 real change requests graded against 2,331 hidden expert criteria.
The best model, OpenAI's GPT-6 Astra, met 69.9 percent of requirements on average but fully solved only 26.4 percent of tasks.
Claude Opus 5 was next, meeting 66.1 percent of criteria and fully solving 18.3 percent of tasks, followed by Qwen3.8-Max. Grok 4.6 was last at 6.6 percent.
GPT-6 Astra fully solved 41.0 percent of localised multimodal tasks but only 17.9 percent of multi-file repository tasks.
The tasks were deliberately weighted toward hard cases that had already caused users trouble, so 26.4 percent is a stress-test number, not the success rate on average coding work.
The paper is an unreviewed preprint, the authors are mostly Alibaba-affiliated, the test used one-shot briefs with no follow-up corrections, and the public dataset is partial, so the result is an early signal, not a final verdict.

Conclusion

Co-Design Bench does not say coding agents are useless. It says they are unfinished: they do most of a job and leave the last part to a human. That is a useful, honest finding, and it matches what working developers already report. Use the agent for speed, keep the human for the finish, and judge every benchmark by asking whose tasks it used and how strict its grading was. On those questions, this new test is one of the best we have, and its 26.4 percent figure deserves to be taken seriously, with its caveats attached.

Search This Blog

INORYVA