If you hand an AI coding agent a real change request, expect to finish the job yourself most of the time. A new benchmark built by researchers at Alibaba and several universities found that the strongest of ten models completely satisfied only 26.4% of its tasks.
The test is called Co-Design Bench. OpenAI’s GPT-6 Astra led it, meeting on average 69.9% of the requirements in each task, but leaving at least one detail undone in nearly three tasks out of four. Anthropic’s Opus 5 managed 18.3% fully solved tasks, Qwen3.8-Max 16.5%, and xAI’s Grok 4.6 came last with 6.6%.
| Model | Tasks fully solved | Requirements met per task |
|---|---|---|
| GPT-6 Astra | 26.4% | 69.9% |
| Opus 5 | 18.3% | 66.1% |
| Qwen3.8-Max | 16.5% | 65.3% |
| Kimi K3 | 15.8% | 62.9% |
| Qwen3.8-27B | 15.0% | 64.4% |
| GPT-5.6 Sol | 14.3% | 61.1% |
| GLM-5.3 Flash | 10.6% | 57.3% |
| Opus 4.8 | 10.3% | 58.1% |
| GLM-5.2 | 9.5% | 56.7% |
| Grok 4.6 | 6.6% | 51.1% |
The paper, posted on 26 October, draws its 273 tasks, 100 on multimodal edits and 173 on whole repositories, from 26,314 real coding sessions, sifted from 37,463 distinct user requests. Each task gives the agent a working web app and asks it to change it, sometimes with an image, audio clip, video or document attached as the brief. Experts wrote 2,331 acceptance criteria between them, and the agents never see them.
Grading is deliberately strict. One AI judge checks the finished app against the criteria using the code and a browser, then a second model tries to refute every pass. A feature counts only if it actually works.
The authors say older tests have run out of room. In the data behind the project page, Opus 4.8 scores 99.4% on an earlier web-editing benchmark, WebCompass-edit, which leaves almost nothing to tell models apart. A model can look near-perfect on a saturated test and still stumble on the messy requests people actually type.
Bigger jobs hurt more. All ten models did worse on repository edits that mixed three or four kinds of change (layout, design, behaviour and content, for example) than on edits mixing one or two. One caveat is that every task is a one-shot brief, so the benchmark says nothing about how agents cope when a user pushes back halfway through.
For anyone choosing a coding assistant, the practical reading is that the demos hide the leftover work. A tool that does most of a job still needs a person to find and fix the missing piece. The benchmark’s public subset, 96 multimodal editing tasks, is on GitHub for anyone who wants to run their own agent through it.
