A fresh run of SlopCodeBench put Claude Opus 5 at a 24% strict pass rate across 17 evolving checkpoints, four times better than Opus 4.8 and Sonnet 5 at 6%, but nowhere near the 97% it scores on SWE-bench Verified. The gap is the difference between fixing one issue and maintaining a codebase as the spec keeps changing.
Read the full story: Opus 5 Scores 97% on SWE-bench and 24% on Slop Code →
Transcript
Claude Opus 5 resolves ninety seven percent of SWE bench verified. That's the number everyone quotes. Run the same model through SlopCodeBench instead, where the spec keeps changing across seventeen checkpoints, and the strict pass rate drops to twenty four percent. Four checkpoints out of seventeen. Not a contradiction, two different jobs. SWE bench hands you one issue and one clean shot at a patch. SlopCodeBench makes you maintain code you already wrote while requirements evolve, carrying every old test forward as a regression check. Opus 5 is a real four times improvement over Opus 4.8 here. But between eighty nine and ninety eight percent of its generated code tripped at least one quality rule, and complexity kept climbing every round. The lesson isn't that the model is bad. A stellar one shot score doesn't mean you can hand it a repo and walk away.