OpenAI retracts SWE-Bench Pro after finding 30% broken

OpenAI published an audit on July 8, 2026, concluding that roughly 30% of the tasks in SWE-Bench Pro are broken, and in the same post it formally retracted its earlier advice that the industry adopt the benchmark. That is a sharp reversal: only months after telling model builders to drop SWE-bench Verified and move to Pro, OpenAI now says Pro cannot be trusted either. Our thesis is blunt: the coding benchmark half the industry quotes just lost its referee, and every leaderboard built on it inherits the doubt.

• Roughly 30% broken. OpenAI's automated pipeline flagged 200 of the 731 public tasks (27.4%) as broken, while a separate campaign of trained human engineers flagged 249 (34.1%). The headline estimate OpenAI settled on is about 30%.
• Recommendation retracted. OpenAI explicitly withdrew its earlier call for model developers to adopt SWE-Bench Pro and advised teams to examine any Pro results carefully.
• Four ways a task breaks. Overly strict tests, underspecified prompts, low-coverage tests, and outright misleading prompts each let a correct fix fail or an incomplete fix pass.
• Agents audited the benchmark. OpenAI ran Codex-based investigator agents alongside five human engineers per flagged task, a preview of models being used to police the evals that grade them.

What did OpenAI actually find?

OpenAI built a data-quality pipeline that reads the instructions given to a model, the model's own attempts, and the hidden tests used to grade them, then flags tasks that look broken. On SWE-Bench Pro's 731-task public split, that filter surfaced 286 suspect tasks. OpenAI then ran two deeper reviews in parallel. Codex-based investigator agents got access to each task repository so they could run tests, inspect files, and separate fair ambiguity from genuine underspecification. Separately, five experienced software engineers reviewed every flagged task, forming an independent judgment from the problem statement, the tests, and the reference solution before seeing any pipeline output.

RelatedGrok 4.5 lands: Opus-class claims, cheaper, unproven

The two paths converged. The agent pipeline labeled 200 tasks (27.4%) broken; the humans labeled 249 (34.1%). Their category judgments overlapped in 74% of cases, and, tellingly, on no flagged task was "not broken" the most common human verdict. Humans were consistently harsher, often tagging a single task with several defects at once, which is why OpenAI frames its own agent-plus-human number as conservative. The gap was widest on low-coverage tests, which humans named the top issue on 9.4% of the benchmark versus 4.1% for the agents.

Why are so many tasks broken?

OpenAI sorts the failures into four buckets. Overly strict tests enforce implementation details the prompt never specified, so a functionally correct solution still fails. Underspecified prompts leave out requirements that the hidden tests silently enforce and that no reasonable engineer could infer. Low-coverage tests under-check the feature, so an incomplete fix slips through as a pass. And a misleading prompt actively points the model at the wrong behavior.

The concrete example OpenAI gives is almost comically small. In an OpenLibrary task, the prompt showed a Markdown table cell formatted with a single leading space, like " | Chapter 1 | 1", while the hidden test demanded two leading spaces. A model that faithfully follows the written prompt produces a one-character difference and is marked wrong. Multiply that kind of mismatch across hundreds of tasks scraped from real pull requests, and the benchmark stops measuring capability and starts measuring luck.

The root cause is structural. These tasks are mined programmatically from open-source issues and merged pull requests, artifacts that were written for humans arguing back and forth, not as clean, implementation-agnostic specs. Tests bundled into a PR are written to validate one specific change, so they naturally over-fit to that change rather than defining a fair standard.

Why does this matter for every AI leaderboard?

Because SWE-Bench Pro was supposed to be the trustworthy successor. When SWE-bench Verified saturated and picked up contamination problems, OpenAI itself told the field to move to Pro, where frontier models had jumped from a 23.3% pass rate to 80.3% in eight months. Vendors took the advice: recent model launches, including this week's Grok 4.5, lead with Pro numbers. If nearly a third of the underlying tasks are broken, then a two or three point gap between two models sits comfortably inside the noise floor, and the ranking is decoration.

The practical fallout: our own AI coding leaderboard and every rival ranking that quotes a Pro score now carries an asterisk. It does not mean the top models are bad. It means the numbers separating them were never as precise as the two-decimal-place charts implied.

RelatedAnthropic's Claude Science targets neglected diseases

Isn't it convenient that OpenAI is the auditor?

It is a fair question, and worth holding in mind. OpenAI has an obvious incentive to discredit any benchmark where it does not cleanly lead, and it is now the second time in a year the company has published a teardown of an eval and told everyone to stop using it. But the evidence here is unusually checkable. The OpenLibrary spacing bug is verifiable by anyone with the repo, five outside engineers reviewed each flagged case, and the failure taxonomy matches complaints benchmark researchers have raised independently. Skepticism about the messenger is healthy; the specific defects are hard to wave away.

What it means for the market

There is no single ticker that moves on a broken benchmark, but the signal for anyone tracking the AI-tooling race is real. Enterprises increasingly pick coding models on published eval scores, and procurement teams cite them in contracts. If Pro is unreliable, the near-term winners are labs with strong real-world adoption and internal evals rather than the best public-benchmark chart, which favors incumbents like Anthropic and OpenAI over challengers leaning on a single flattering number. Watch how quickly vendors stop quoting Pro in launch decks; that is the tell that the retraction landed.

What happens to coding benchmarks now?

OpenAI's proposed fix is to have experienced software developers build new benchmarks from scratch, designed as clean tasks rather than scraped from messy PR histories, with human oversight throughout. In the meantime there is a vacuum: no single public coding eval currently carries consensus trust. The more durable shift may be the method itself. OpenAI just demonstrated that frontier models are now good enough to audit the datasets that grade them, at a scale and depth that was impractical a year ago. Expect eval hygiene, models checking benchmarks before anyone reports a score, to become standard practice.

Our take

This is the most important AI-eval story of the month precisely because it is boring plumbing rather than a shiny new model. For two years the industry has argued about coding leaderboards as if the underlying rulers were exact. OpenAI just showed that one of the most-quoted rulers is bent on roughly a third of its markings, and had the awkward honesty to retract its own prior advice. The uncomfortable takeaway is that a lot of recent "Model A beats Model B on Pro" coverage, some of it ours, was measuring noise. The healthier future is one where nobody reports a benchmark number without first auditing the benchmark, and where the models themselves do the auditing. That is genuinely new, and it is more consequential than any single score it will eventually replace.

Original analysis by GenZTech. Primary source: OpenAI.