head-to-head
| Metric | GPT-5.6 Luna | Claude Opus 4.8 |
|---|---|---|
| SWE-bench Verified | 93.0% | 88.6% |
| SWE-bench Pro | — | 69.2% |
| Terminal-Bench | — | ~82.7% (TB2.1) |
| Input $ / 1M | — | $5 |
| Output $ / 1M | — | $25 |
| Context | — | 1M |
| Open weights | No | No |
| Access | API | API · Claude Code · claude.ai (Max) |
| Maker | OpenAI | Anthropic |
what do the benchmarks actually say?
On SWE-bench Verified — real, human-validated GitHub issues resolved end-to-end — GPT-5.6 Luna posts 93.0% against 88.6% for Claude Opus 4.8, a 4.4-point gap. Verified is the closest public proxy for "can it fix a real bug in a real repo without help", which is why it anchors our ranking.
A few points either way is real but not decisive: within that band, the agent scaffolding around the model — how it retrieves files, runs tests, and retries — often matters as much as the base model. Treat the gap as a lean, not a verdict.
which is cheaper to run?
Public per-token pricing isn't confirmed for both models, so we don't print a cost comparison yet.
when to pick each
The cost outlier of the leaders: within striking distance of the top scores while costing a fraction per solved task, which makes it the obvious candidate for high-volume agent runs.
The hardest agentic refactors and long, autonomous multi-file tasks where every point of accuracy saves a human review cycle.
how were these scores verified?
We only print a number once it's confirmed against a primary source or an independent evaluation, and each row on our leaderboard records which kind it is:
- GPT-5.6 Luna: Independent (vals.ai, eval listed Jul 17 2026, mini-swe-agent bash-only harness): SWE-bench Verified 93.00% ±1.14. Added Jul 21, 2026 — this row was missing from the board even though vals.ai had already evaluated Luna, and our Kimi K3 note referenced its 93.0% score without ever listing it; adding it moves every row below it down one rank. Treat 3rd and 4th as a tie: Kimi K3's 93.40% ±1.11 is 0.4 points higher, well inside the combined margin of error (~0.25 sigma), so the ordering between them is not significant. Like the rest of the GPT-5.6 family, OpenAI has published no SWE-bench Verified figure of its own, so we rank on the independent number per our standing rule. The striking number is cost: vals.ai measured $0.21 per test against $1.15 for GPT-5.6 Sol, $1.92 for Claude Opus 4.8 and $2.05 for Claude Fable 5, at 201s median latency. Per-token list pricing not confirmed against OpenAI's own pricing page, so inPrice/outPrice stay blank rather than estimated.
- Claude Opus 4.8: Independent (vals.ai, Jul 14 2026, mini-swe-agent bash-only harness): SWE-bench Verified 88.6% ±1.42. Corrected Jul 17, 2026: we previously printed Anthropic's own "~86%" and claimed independent evals tracked it within ~1 point, which was wrong — the independent number is 2.6 points higher, and our methodology is to prefer the independent one. Run through the Claude Code harness instead of the bare bash agent, vals.ai measures 85.8%, a reminder that the harness moves these numbers as much as the model does. SWE-bench Pro 69.2% is Anthropic-reported.
Ranked on our AI Coding Leaderboard, updated 2026-07-21. Scores are confirmed against primary sources; prices are per 1M input tokens and can change.
- OpenAIvals.ai — SWE-bench Verified (independent) — Independent (vals.ai, eval listed Jul 17 2026, mini-swe-agent bash-only harness): SWE-bench Verified 93.00% ±1.14. Added Jul 21, 2026 — this row was missing from the board even though vals.ai had already evaluated Luna, and our Kimi K3 note referenced its 93.0% score without ever listing it; adding it moves every row below it down one rank. Treat 3rd and 4th as a tie: Kimi K3's 93.40% ±1.11 is 0.4 points higher, well inside the combined margin of error (~0.25 sigma), so the ordering between them is not significant. Like the rest of the GPT-5.6 family, OpenAI has published no SWE-bench Verified figure of its own, so we rank on the independent number per our standing rule. The striking number is cost: vals.ai measured $0.21 per test against $1.15 for GPT-5.6 Sol, $1.92 for Claude Opus 4.8 and $2.05 for Claude Fable 5, at 201s median latency. Per-token list pricing not confirmed against OpenAI's own pricing page, so inPrice/outPrice stay blank rather than estimated.
- Anthropicvals.ai — SWE-bench Verified (independent) — Independent (vals.ai, Jul 14 2026, mini-swe-agent bash-only harness): SWE-bench Verified 88.6% ±1.42. Corrected Jul 17, 2026: we previously printed Anthropic's own "~86%" and claimed independent evals tracked it within ~1 point, which was wrong — the independent number is 2.6 points higher, and our methodology is to prefer the independent one. Run through the Claude Code harness instead of the bare bash agent, vals.ai measures 85.8%, a reminder that the harness moves these numbers as much as the model does. SWE-bench Pro 69.2% is Anthropic-reported.
- BenchmarkSWE-bench — the real-GitHub-issue benchmark