Opus 5 Scores 97% on SWE-bench and 24% on Slop Code

Claude Opus 5 resolves 97.0% of SWE-bench Verified on a neutral bash-only harness, the top score on our AI coding leaderboard. Run the same model against SlopCodeBench, where the requirements keep changing under it, and the strict pass rate drops to 24%. Four checkpoints out of seventeen. That is not a contradiction between two benchmarks so much as a measurement of two genuinely different jobs, and only one of them is what most engineering work actually looks like.

The run comes from a public evaluation published this week by the team behind the advanced-context-engineering-for-coding-agents project, which took three problems of varying difficulty from SlopCodeBench, generated 17 checkpoints, and put three Claude models through each one with fresh context and identical prompts.

RelatedClaude Sonnet 5: Near-Opus Coding at Half the Price

What SlopCodeBench measures that SWE-bench does not

SlopCodeBench came out of UW Madison in March 2026 with a specific complaint about how coding models are evaluated: real problems do not arrive fully specified. The benchmark hands a model a partial spec, takes the solution, then extends the requirements at the next checkpoint. The model has to modify what it already wrote rather than starting clean, and the tests it passed earlier come along as inherited regression tests. Strict pass means everything passes, including the old tests.

SWE-bench Verified, by contrast, gives a model one well-scoped GitHub issue and one shot at a patch. It measures whether the model can read a codebase and land a fix. It says nothing about what that codebase looks like after the model has touched it eight times.

The numbers

Two things stand out. Opus 5 is a real improvement, four times Opus 4.8 on the same 17 checkpoints, and the gain is not an artefact of a higher SWE-bench score, since Opus 4.8 and Sonnet 5 tie at 6% despite a nine-point spread on SWE-bench. But 17 checkpoints across three problems is a small sample, and the earlier paper's numbers came from a different selection, so treat the cross-table comparison as directional rather than a like-for-like ranking.

What happens to the code along the way

The pass rate is only half of it. The evaluation also ran 41 deterministic quality measures across the generated code, covering cyclomatic complexity, duplication, code volume, decomposition and dependency structure. Between 89% and 98% of the generated code tripped at least one quality rule. Complexity climbed checkpoint over checkpoint, duplication grew, and Opus 5 in particular wrote roughly five times more functions than the other two models.

That last detail is worth reading carefully, because more functions is not automatically worse. Decomposition is usually a virtue. But paired with rising duplication it looks less like refactoring and more like the model reaching for a new helper instead of finding the one it wrote two checkpoints ago. None of the three models reached a final checkpoint without accumulated defects.

Defect accumulation is the mechanism the benchmark is really probing. A failure at checkpoint three does not stay at checkpoint three. It comes back as a broken inherited regression test at four, five and six, which is why strict pass rates collapse in a way that per-task rates do not.

RelatedGPT-5.6 Sol's #1 coding score is basically a tie

What this means for the market

The commercial pitch for coding agents has been drifting from "helps you write a patch" toward "runs unsupervised on your backlog," and that second claim is priced very differently. This evaluation is a data point against the unsupervised version. If the best available model finishes fewer than one in four evolving tasks cleanly, the product that sells is still the reviewed one, which caps how far seat-based coding tools can move to outcome-based pricing.

The signal for anyone budgeting AI engineering spend is that the benchmark you buy on matters. A 97% SWE-bench headline is real and is not a proxy for maintenance quality, and the cost that actually shows up later is review time on code that got measurably more complex every round. Watch whether any vendor starts publishing a maintenance-style score voluntarily. Nobody does yet.

Our take

This is the most useful counterweight to a benchmark score we have seen this month, and it should not be read as debunking anything. Opus 5 quadrupling its predecessor on the harder task is a genuine result, and it happened alongside the SWE-bench gain rather than instead of it.

What it punctures is the inference people keep drawing from a 97%: that a model this good at resolving issues can be handed a repository and left alone. Fewer than one in four evolving tasks completed cleanly, with quality violations in nearly all generated code, says the opposite. The models are excellent at the thing SWE-bench measures and we should keep measuring it. We should also stop pretending it measures the other thing. Sustained ownership of a codebase, where yesterday's shortcut becomes tomorrow's failing test, is still a human job with a very fast assistant.

Original analysis by GenZTech, comparing a public SlopCodeBench evaluation against independently measured SWE-bench Verified scores. Source: github.com/humanlayer