{
  "name": "GENZ TECH AI Coding Leaderboard",
  "description": "AI coding models ranked by SWE-bench Verified (tiebreak SWE-bench Pro), with each model's vendor-reported and independently re-run SWE-bench Verified figures side by side, Terminal-Bench, and API pricing. Models with no SWE-bench Verified score are listed unranked.",
  "url": "https://genztech.blog/ai-coding-leaderboard/",
  "updated": "2026-09-28",
  "updatedMeans": "The date the ranked data last actually changed, derived from this dataset's own revision history — not the date the file was last edited.",
  "sourcesChecked": "2026-10-16",
  "sourcesCheckedMeans": "The date we last re-ran the unranked models against an independent evaluation. Moves even when nothing changes; if it equals or trails `updated`, the board simply had no movement to report.",
  "independentArchived": "2026-09-05",
  "independentArchivedMeans": "The date vals.ai archived its SWE-bench Verified board. No new independent scores arrive from that source after it, so later models may stay unranked.",
  "metric": "SWE-bench Verified",
  "license": "https://creativecommons.org/licenses/by/4.0/",
  "attribution": "Data by GENZ TECH (https://genztech.blog/ai-coding-leaderboard/). Free to cite with attribution + a link.",
  "methodology": "We rank by SWE-bench Verified (500 real, human-validated GitHub issues resolved end-to-end), tiebroken by the harder SWE-bench Pro. A score is only printed once confirmed against an independent evaluation or the maker's primary source — and every row states which kind it is. Where both exist, we print both: one as the ranked score, the other beside it in the table and in that row's note. We would rather show you the gap than ask you to trust our pick. Our independent reference is vals.ai, which runs every model itself through the same minimal bash-only harness (mini-swe-agent), so the models are compared on equal footing. That matters more than it sounds: SWE-bench scores a model and its scaffolding together, and vendors report using their own tuned scaffolds. Against vals.ai's neutral harness, the vendor claims on this board usually run optimistic, by 4.0 to 11.6 points, though two run below the independent figure; the vendor-versus-independent chart on this page plots every pair. So rows marked vendor-reported are best-case numbers and are not strictly comparable to the independent ones — where we know the independent figure, we print it beside the score and in the row's note. That is a deliberate choice and you should know we made it: on five rows (Claude Sonnet 5, MiniMax M3, Qwen3.7 Max, Kimi K2.6, DeepSeek V4 Pro) an independent score for that exact model exists and disagrees with the maker's, and we still rank on the maker's published figure because it is the number that model is sold and quoted on. On four of those the independent number is lower. DeepSeek V4 Pro runs the other way: vals.ai measures the 0813 GA build 15.8 points ABOVE DeepSeek's own claim, which the disclose-don't-rebase rule was never written for, so that row is flagged in its note for a human decision on whether to rebase it. Until that decision is made it stays ranked on DeepSeek's 80.6%, which is the position the table shows, with vals.ai's 96.4% printed beside it. We disclose the independent one beside it instead of quietly restating the board on a single evaluator's harness choice. The honest consequence: positions that straddle the two regimes are approximate. Qwen3.7 Max is the sharpest case: it is ranked on Alibaba's 80.4%, and on the neutral harness its 68.8% would put it below every other ranked model. Note that llm-stats, which we previously miscredited as an independent tracker, labels its own SWE-bench Verified table \"Verified: 0 / Self-reported: 104\"; it aggregates vendor claims. Models with no SWE-bench Verified score from either the maker or an independent evaluator are not ranked and not estimated. They are listed separately as the newest models not yet independently ranked, with any figures their makers published on other benchmarks (Terminal-Bench, SWE-bench Pro) labelled vendor-reported, because those numbers are not comparable to the ranked ones. vals.ai archived its SWE-bench Verified board on 5 September 2026, so some of these models may never receive an independent score, and we will not move them into the ranking on a different benchmark's number. A model whose maker published a SWE-bench Verified score that no independent evaluator has re-run (Ember-1 and Gemini 3.1 Pro today) is ranked on that figure and labelled vendor-reported in the table. Prices are per 1M input tokens on the standard API tier and can change — always confirm current pricing with the provider.",
  "fields": {
    "sweBenchVerified": "The ranked SWE-bench Verified score. Equals either sweBenchVerifiedVendor or sweBenchVerifiedIndependent; rankedOn says which.",
    "sweBenchVerifiedVendor": "The maker's own published SWE-bench Verified figure for this exact model/build, on the maker's scaffold. Null when the maker published none.",
    "sweBenchVerifiedIndependent": "An independent re-run of the same model/build (independentSource, on independentDate). Null when none exists.",
    "rankedOn": "\"vendor\" or \"independent\": which of the two figures the rank uses. Null for unranked models.",
    "announced": "Unranked models only: the date the maker announced or released the model.",
    "note": "Full provenance note for the row, as shown on the model's page."
  },
  "models": [
    {
      "rank": 1,
      "name": "Claude Opus 5",
      "maker": "Anthropic",
      "open": false,
      "sweBenchVerified": "97.0%",
      "sweBenchVerifiedVendor": null,
      "sweBenchVerifiedIndependent": "97.0%",
      "independentSource": "vals.ai",
      "independentDate": "2026-07-25",
      "rankedOn": "independent",
      "sweBenchPro": null,
      "terminalBench": null,
      "inputPricePer1M": "$5",
      "outputPricePer1M": "$25",
      "context": null,
      "verified": true,
      "announced": null,
      "source": {
        "title": "vals.ai — SWE-bench Verified (independent)",
        "url": "https://www.vals.ai/benchmarks/swebench"
      },
      "note": "Independent (vals.ai, observed Jul 25 2026, mini-swe-agent bash-only harness): SWE-bench Verified 97.00% ±0.76, the highest score on the board and 1st of the 75 systems vals.ai has run. Entered ranked Jul 25, 2026 after one day unranked: Anthropic published no SWE-bench Verified number at launch and still has not, so this is vals.ai's own measurement rather than a vendor claim. Read the #1 as a three-way tie, not a win — GPT-5.6 Sol is at 96.20% ±0.86 (a 0.8-point gap, ~0.7 sigma) and Claude Fable 5 at 95.00% ±0.98 (2.0 points, ~1.6 sigma), both inside the combined margin of error. We rank Opus 5 first only because it scored highest. Where the top two genuinely separate is task length, and not in Opus 5's favour: on the 1-to-4-hour tier Sol solves 98% against Opus 5's 90%, while Opus 5 leads on shorter work (98% under 15 minutes and 97% on 15-minute-to-1-hour tasks, vs 97% and 95% for Sol). Released Jul 24, 2026 at $5/$25 per 1M, the same price as Opus 4.8 and half of Fable 5. Anthropic's launch claims stay unreproducible (Frontier-Bench v0.1, CursorBench 3.2 and Zapier AutomationBench are proprietary), so this is the first externally checkable score the model has. Fast mode runs about 2.5x default speed at 2x base price.",
      "sameAs": []
    },
    {
      "rank": 2,
      "name": "GPT-5.6 Sol",
      "maker": "OpenAI",
      "open": false,
      "sweBenchVerified": "96.2%",
      "sweBenchVerifiedVendor": null,
      "sweBenchVerifiedIndependent": "96.2%",
      "independentSource": "vals.ai",
      "independentDate": "2026-07-14",
      "rankedOn": "independent",
      "sweBenchPro": null,
      "terminalBench": "88.8%",
      "inputPricePer1M": "$5",
      "outputPricePer1M": "$30",
      "context": null,
      "verified": true,
      "announced": null,
      "source": {
        "title": "vals.ai — SWE-bench Verified (independent)",
        "url": "https://www.vals.ai/benchmarks/swebench"
      },
      "note": "Independent (vals.ai, Jul 14 2026, mini-swe-agent bash-only harness): SWE-bench Verified 96.20% ±0.86 — now the second-highest score on the board. Correction, Jul 25, 2026: this row read \"the top score on the board\" from Jul 17 until Jul 25, when vals.ai evaluated Claude Opus 5 at 97.00% ±0.76 and took the #1 slot. The 0.8-point gap is ~0.7 sigma and not significant, so the two are a statistical tie, and Sol still leads on the longest tasks (98% vs 90% on the 1-to-4-hour tier). Verified Jul 17, 2026; it had been unranked since Jun 26 because OpenAI published no SWE-bench number of its own, and it still has not. Read the #1 with care: the 1.2-point lead over Claude Fable 5 (95.00% ±0.98) is inside the combined margin of error (~0.9 sigma, not significant), so the two are a statistical tie and we rank Sol first only because it scored higher. Where it does separate is task length — 98% on 1-4 hour tasks vs 93% for Fable 5. OpenAI's own Terminal-Bench 2.1 claim is 88.8% (Sol) / 91.9% (Sol Ultra). No SWE-bench Pro score published. Pricing $5/$30 per 1M.",
      "sameAs": []
    },
    {
      "rank": 3,
      "name": "Grok 4.6",
      "maker": "SpaceXAI (xAI)",
      "open": false,
      "sweBenchVerified": "95.6%",
      "sweBenchVerifiedVendor": null,
      "sweBenchVerifiedIndependent": "95.6%",
      "independentSource": "vals.ai",
      "independentDate": "2026-08-12",
      "rankedOn": "independent",
      "sweBenchPro": null,
      "terminalBench": null,
      "inputPricePer1M": "$2",
      "outputPricePer1M": "$6",
      "context": null,
      "verified": true,
      "announced": null,
      "source": {
        "title": "vals.ai — SWE-bench Verified (independent)",
        "url": "https://www.vals.ai/benchmarks/swebench"
      },
      "note": "Independent (vals.ai, benchmark updated 2026-08-12, mini-swe-agent bash-only harness): SWE-bench Verified 95.60% ±0.92. Entered ranked 2026-08-13, one day after launch, and it enters near the top: 4th of the 82 systems vals.ai has run, behind Claude Opus 5 (97.00% ±0.76), DeepSeek V4 Pro 0813 (96.40% ±0.83) and GPT-5.6 Sol (96.20% ±0.86). Read the gap to Claude Fable 5 below it (95.00% ±0.98) as a tie: 0.6 points against a pooled SEM of about 1.34 is well inside the margin of error, and we rank Grok 4.6 higher only because it scored higher. Same against Sol above it, a 0.6-point gap. This is the second time running that xAI shipped a Grok with no SWE-bench Verified number of its own and vals.ai supplied one: Grok 4.5 waited nine days, Grok 4.6 waited one. SpaceXAI still publishes no SWE-bench Verified or SWE-bench Pro figure, so the ranked score here is entirely vals.ai's measurement with no vendor claim to disclose against. What xAI did publish, all vendor-reported on its own table: AA Intelligence Index 61, CursorBench v3.2 69.9%, DeepSWE v1.1 65.9%, FrontierCode v1.1 Extended 61.3%, APEX-Agents 57.5%, APEX-SWE 56.4%, Terminal-Bench v3.0 26%, GDPVal-AA v2 1753, AA-Briefcase 1577, Harvey LAB (Vals) 15.8%. That table's shape still holds and is worth reading against this score: Grok 4.6 takes the best result on the three knowledge-work evals and loses the two hardest agentic-coding ones by wide margins (DeepSWE 65.9% vs Sol's 73%, Terminal-Bench v3.0 26% vs Sol's 34.6%). Terminal-Bench v3.0 is NOT comparable to the 83.3% TB2.1 figure on the Grok 4.5 row: on v3.0 Grok 4.5 scores 15.7% and the whole field tops out near 34%. On cost it is the standout of the top five, $0.78 per test against $1.29 for Opus 5, $1.15 for Sol and $2.05 for Fable 5, though it is also the slowest of them at 604s. Pricing unchanged from Grok 4.5 at $2/$6 per 1M. See /p/grok-4-6-frontier-benchmarks-terminal-bench-gap/.",
      "sameAs": []
    },
    {
      "rank": 4,
      "name": "GPT-5.6 Terra",
      "maker": "OpenAI",
      "open": false,
      "sweBenchVerified": "95.4%",
      "sweBenchVerifiedVendor": null,
      "sweBenchVerifiedIndependent": "95.4%",
      "independentSource": "vals.ai",
      "independentDate": "2026-08-20",
      "rankedOn": "independent",
      "sweBenchPro": null,
      "terminalBench": null,
      "inputPricePer1M": "$2",
      "outputPricePer1M": "$12",
      "context": "1M",
      "verified": true,
      "announced": null,
      "source": {
        "title": "vals.ai — SWE-bench Verified (independent)",
        "url": "https://www.vals.ai/benchmarks/swebench"
      },
      "note": "Independent (vals.ai, re-evaluated, mini-swe-agent bash-only harness): SWE-bench Verified 95.4% ±0.94, 5th of 86 systems. CORRECTION, 2026-08-20: this row previously carried 75.2% ±1.93 (32nd of 75), the figure vals.ai published on Jul 22 2026 and which we added Jul 27, 2026. A routine sweep found vals.ai had re-run the model and revised the score up by 20.2 points; the old figure and the reading we built on it, that Terra was the weakest of the three GPT-5.6 tiers, are both withdrawn. A jump that large on an unchanged model points at the first run rather than the model, so read 95.4% as the corrected measurement and not as an improvement. Terra now sits second among the GPT-5.6 tiers, behind Sol at 96.2% and ahead of Luna at 93.0%. Treat it as tied with GLM-5.3 (also 95.4% ±0.94), Grok 4.6 (95.6%) and Claude Fable 5 (95.0%): pooled SEM across those rows is about 1.3 points and every gap is smaller than that. Cheapest of the 95%-plus cluster on measured cost per test at $0.40, against $1.15 for Sol and $2.05 for Fable 5. Repriced Aug 1, 2026: OpenAI cut Terra 20% on Jul 30, 2026, from $2.50/$15 to $2/$12 per 1M (vendor-announced, reported by BleepingComputer and VentureBeat). Cached input $0.25 at launch, not re-confirmed after the cut. Context ~1.05M, max output 128k. Released Jul 9, 2026.",
      "sameAs": []
    },
    {
      "rank": 5,
      "name": "GLM-5.3",
      "maker": "Z.ai (Zhipu AI)",
      "open": false,
      "sweBenchVerified": "95.4%",
      "sweBenchVerifiedVendor": null,
      "sweBenchVerifiedIndependent": "95.4%",
      "independentSource": "vals.ai",
      "independentDate": "2026-08-20",
      "rankedOn": "independent",
      "sweBenchPro": null,
      "terminalBench": "28.3 (TB3.0, vendor)",
      "inputPricePer1M": null,
      "outputPricePer1M": null,
      "context": "1M",
      "verified": true,
      "announced": null,
      "source": {
        "title": "vals.ai — SWE-bench Verified (independent)",
        "url": "https://www.vals.ai/benchmarks/swebench"
      },
      "note": "Independent (vals.ai, 2026-08-20 sweep, mini-swe-agent bash-only harness): SWE-bench Verified 95.4% ±0.94, 6th of 86 systems. Left the verifying queue on 2026-08-20, six days after launch. Z.ai published NO SWE-bench Verified figure of its own, so this rank rests entirely on the independent run, which is the cleanest kind of row on this board. Read it as tied with GPT-5.6 Terra (95.4% ±0.94), Grok 4.6 (95.6%) and Claude Fable 5 (95.0%); pooled SEM is about 1.3 points and every gap is inside it. That result is striking given the release is post-training only, on the same base model as GLM 5.2, which is ranked far below on an independent 82.8%. Vendor numbers from the Aug 14 2026 release post, run mostly inside a Claude Code 2.1.207 harness at max effort rather than a neutral one: Terminal-Bench 3.0 28.3 (up from GLM-5.2’s 4.6), DeepSWE v1.1 66.9 (from 46.2), Agents’ Last Exam 28.5 (from 23.8). Its headline claims were cyber rather than coding: CyberGym 84.5% and ExploitBench 54.4%. Marked closed because the weights are NOT out: Z.ai said roughly two weeks after launch pending safety hardening, with no license announced, so the open-weights flag flips only when they actually land. See /p/glm-5-3-cybergym-open-weights-delayed/.",
      "sameAs": []
    },
    {
      "rank": 6,
      "name": "Claude Fable 5",
      "maker": "Anthropic",
      "open": false,
      "sweBenchVerified": "95.0%",
      "sweBenchVerifiedVendor": null,
      "sweBenchVerifiedIndependent": "95.0%",
      "independentSource": "vals.ai",
      "independentDate": "2026-07-14",
      "rankedOn": "independent",
      "sweBenchPro": "80.3%",
      "terminalBench": null,
      "inputPricePer1M": "$10",
      "outputPricePer1M": "$50",
      "context": "1M",
      "verified": true,
      "announced": null,
      "source": {
        "title": "GENZ TECH — Claude Fable 5 returns",
        "url": "https://genztech.blog/p/claude-fable-5-returns-retakes-coding-crown/"
      },
      "note": "Independent (vals.ai, Jul 14 2026, mini-swe-agent bash-only harness): SWE-bench Verified 95.00% ±0.98. Held the top score until GPT-5.6 Sol was evaluated at 96.20% ±0.86, and now sits third behind Claude Opus 5 (97.00% ±0.76, added Jul 25 2026). All three are within the combined margin of error of each other. Against Sol it is a 1.2-point gap — a 1.2-point gap that is inside the combined margin of error (~0.9 sigma), so the two are a statistical tie and we rank Sol first only because it scored higher. Re-confirmed 2026-08-16: vals.ai still measures 95.00% ±0.98, unchanged across three sweeps (Aug 13, 15, 16), and Grok 4.6 sits above it at 95.60% ±0.92, a 0.6-point gap that is inside the combined margin of error and should be read as a tie. SWE-bench Pro 80.3% uses Anthropic's own scaffolding and is contested. Restored Jul 1, 2026 after a 20-day export-control suspension. Pricing $10/$50 per 1M.",
      "sameAs": [
        "https://www.anthropic.com/claude"
      ]
    },
    {
      "rank": 7,
      "name": "Kimi K3",
      "maker": "Moonshot AI",
      "open": true,
      "sweBenchVerified": "93.4%",
      "sweBenchVerifiedVendor": null,
      "sweBenchVerifiedIndependent": "93.4%",
      "independentSource": "vals.ai",
      "independentDate": "2026-07-18",
      "rankedOn": "independent",
      "sweBenchPro": null,
      "terminalBench": "88.3%",
      "inputPricePer1M": "$3",
      "outputPricePer1M": "$15",
      "context": "1M",
      "verified": true,
      "announced": null,
      "source": {
        "title": "vals.ai — SWE-bench Verified (independent)",
        "url": "https://www.vals.ai/benchmarks/swebench"
      },
      "note": "Independent (vals.ai, mini-swe-agent bash-only harness): SWE-bench Verified 93.40% ±1.11. Verified Jul 18, 2026 — vals.ai had not run K3 at our Jul 17 check and now ranks it third overall (above GPT-5.6 Luna at 93.0% and Claude Opus 4.8 at 88.6%), making it the highest-scoring open-weight coder we track. Moonshot published no SWE-bench Verified score of its own, only Terminal-Bench 2.1 88.3% on its KimiCode harness, so we rank on the independent number per our standing rule. Corroborated by a second independent evaluator: Artificial Analysis, which runs its own tests, has K3 4th on GDPval-AA v2 (Elo 1684), 2nd on AA-Briefcase (Elo 1545) and 4th of 187 on its Intelligence Index v4.1 (57.1). It also classifies K3 as proprietary outright. AA measured Terminal-Bench 2.1 at 85.02% against Moonshot's own claim of 88.3%, a 3.3-point vendor overstatement, which is why we print the independent figure. Blind developer voting on Arena WebDev ranks it 1st of 99. Price confirmed on Moonshot's own pricing page: $3/$15 per 1M, cache-hit input $0.30 (platform.kimi.ai, checked Jul 19, 2026). **open: false, corrected Jul 19, 2026** — we had this flagged as open weights, which put it top of our \"best open-source coding model\" page as a model nobody can download. Checked Jul 19: no K3 repo under huggingface.co/moonshotai (18 repos, none K3) and github.com/MoonshotAI/Kimi-K3 returns 404. Moonshot's own blog still says \"The full model weights will be released by July 27, 2026.\" Until they actually ship, open is a roadmap item, not a fact, and this row is an API model. Flip back to open: true when the weights land. Re-checked Jul 27, 2026 (Moonshot's own deadline day): huggingface.co/moonshotai still lists no K3 repo, newest is Kimi-K2.7-Code. Score unchanged at 93.40% on vals.ai. **open: true, restored Jul 27, 2026 13:31 UTC**: the weights actually landed at huggingface.co/moonshotai/Kimi-K3 later the same day, 96 safetensors shards, 1,560,998,983,759 bytes (about 1,454 GiB), under a bespoke \"Kimi K3 License\" that is permissive but not Apache 2.0 (separate agreement required above $20M MaaS revenue; attribution required above 100M MAU). Verified against the Hugging Face API and config.json, not a press report. One caveat that belongs on the record: the release is 4-bit MXFP4 only for the routed experts (quantization-aware trained from the SFT stage, format mxfp4-pack-quantized, group size 32), with attention, shared experts, LM head and vision tower left at bf16, and there is no base checkpoint, so this is open for inference rather than open for training. Score unchanged at 93.40%; nothing was re-ranked by this flag. See /p/kimi-k3-weights-landed-4-bit-only/.",
      "sameAs": []
    },
    {
      "rank": 8,
      "name": "GPT-5.6 Luna",
      "maker": "OpenAI",
      "open": false,
      "sweBenchVerified": "93.0%",
      "sweBenchVerifiedVendor": null,
      "sweBenchVerifiedIndependent": "93.0%",
      "independentSource": "vals.ai",
      "independentDate": "2026-07-17",
      "rankedOn": "independent",
      "sweBenchPro": null,
      "terminalBench": null,
      "inputPricePer1M": "$0.20",
      "outputPricePer1M": "$1.20",
      "context": null,
      "verified": true,
      "announced": null,
      "source": {
        "title": "vals.ai — SWE-bench Verified (independent)",
        "url": "https://www.vals.ai/benchmarks/swebench"
      },
      "note": "Independent (vals.ai, eval listed Jul 17 2026, mini-swe-agent bash-only harness): SWE-bench Verified 93.00% ±1.14. Added Jul 21, 2026 — this row was missing from the board even though vals.ai had already evaluated Luna, and our Kimi K3 note referenced its 93.0% score without ever listing it; adding it moves every row below it down one rank. Treat 3rd and 4th as a tie: Kimi K3's 93.40% ±1.11 is 0.4 points higher, well inside the combined margin of error (~0.25 sigma), so the ordering between them is not significant. Like the rest of the GPT-5.6 family, OpenAI has published no SWE-bench Verified figure of its own, so we rank on the independent number per our standing rule. The striking number is cost: vals.ai measured $0.21 per test against $1.15 for GPT-5.6 Sol, $1.92 for Claude Opus 4.8 and $2.05 for Claude Fable 5, at 201s median latency. Pricing added Aug 1, 2026: OpenAI cut Luna 80% on Jul 30, 2026, from $1/$6 to $0.20/$1.20 per 1M, making it the cheapest ranked model on this board per token. Vendor-announced (OpenAI, via its own post and consistent reporting from BleepingComputer and VentureBeat); openai.com/api/pricing returns 403 to our fetcher, so this is a vendor figure rather than one we read off the pricing page ourselves. It had been left blank since Jul 21 for exactly that reason.",
      "sameAs": []
    },
    {
      "rank": 9,
      "name": "Ember-1",
      "maker": "Fireworks AI (built on Moonshot Kimi K3)",
      "open": false,
      "sweBenchVerified": "92.2%",
      "sweBenchVerifiedVendor": "92.2%",
      "sweBenchVerifiedIndependent": null,
      "independentSource": null,
      "independentDate": null,
      "rankedOn": "vendor",
      "sweBenchPro": null,
      "terminalBench": "82.0% (TB2.1, vendor)",
      "inputPricePer1M": null,
      "outputPricePer1M": null,
      "context": null,
      "verified": true,
      "announced": null,
      "source": {
        "title": "Fireworks AI: Introducing Ember-1 (vendor)",
        "url": "https://fireworks.ai/blog/ember-1"
      },
      "note": "Vendor-reported (Fireworks, Sept 23 2026, Fireworks harness): SWE-bench Verified 92.2% using 15.5% fewer tokens than Kimi K3; Terminal Bench 2.1 82.0% (-51.9% tokens vs K3 Max); DeepSWE 1.1 75.2% (-23.7% tokens); live A/B tests 35% fewer tokens per task. No independent score: vals.ai archived its SWE-bench Verified board on 2026-09-05 and it does not list Ember-1 (checked 2026-09-28). Its base, Kimi K3, measures 93.4% independently on vals.ai, so treat the 1.2-point gap as indicative only, since the two figures come from different harnesses. Pricing not stated in the announcement. See /p/fireworks-ember-1-kimi-k3-fewer-reasoning-tokens/.",
      "sameAs": []
    },
    {
      "rank": 10,
      "name": "DeepSeek V4 Flash",
      "maker": "DeepSeek",
      "open": true,
      "sweBenchVerified": "88.8%",
      "sweBenchVerifiedVendor": null,
      "sweBenchVerifiedIndependent": "88.8%",
      "independentSource": "vals.ai",
      "independentDate": "2026-08-05",
      "rankedOn": "independent",
      "sweBenchPro": null,
      "terminalBench": "82.7 (TB2.1, vendor)",
      "inputPricePer1M": "$0.14",
      "outputPricePer1M": "$0.28",
      "context": "1M",
      "verified": true,
      "announced": null,
      "source": {
        "title": "vals.ai — SWE-bench Verified (independent)",
        "url": "https://www.vals.ai/benchmarks/swebench"
      },
      "note": "Independent (vals.ai, Aug 5 2026, mini-swe-agent bash-only harness): SWE-bench Verified 88.8%, for the DeepSeek-V4-Flash-0731 build, matching the checkpoint DeepSeek own change log describes. Ranks above Claude Opus 4.8 (88.6%) and below GPT-5.6 Luna (93.0%). Vendor-reported agent numbers for the same build (DeepSeek change log, Jul 31 2026): Terminal-Bench 2.1 82.7, Cybergym 76.7, Toolathlon verified 70.3, DeepSWE 54.4, NL2Repo 54.2, Agent Last Exam 25.2, produced with DeepSeek own unreleased \"DeepSeek Harness minimal mode\" and not independently reproduced. The V4-Flash-Preview model card (pre-0731 weights) separately listed SWE-bench Verified 79.0 and Terminal-Bench 2.0 56.9; those are not carried over. Same architecture and size as the preview (284B total / 13B active, FP4+FP8), re-post-trained only. Not to be confused with DeepSeek V4 Pro (ranked, 80.6%) or the plain DeepSeek V4 (77.4%). See /p/deepseek-v4-flash-claims-82-7-terminal-bench/.",
      "sameAs": []
    },
    {
      "rank": 11,
      "name": "Claude Opus 4.8",
      "maker": "Anthropic",
      "open": false,
      "sweBenchVerified": "88.6%",
      "sweBenchVerifiedVendor": "~86%",
      "sweBenchVerifiedIndependent": "88.6%",
      "independentSource": "vals.ai",
      "independentDate": "2026-07-14",
      "rankedOn": "independent",
      "sweBenchPro": "69.2%",
      "terminalBench": "~82.7% (TB2.1)",
      "inputPricePer1M": "$5",
      "outputPricePer1M": "$25",
      "context": "1M",
      "verified": true,
      "announced": null,
      "source": {
        "title": "vals.ai — SWE-bench Verified (independent)",
        "url": "https://www.vals.ai/benchmarks/swebench"
      },
      "note": "Independent (vals.ai, Jul 14 2026, mini-swe-agent bash-only harness): SWE-bench Verified 88.6% ±1.42. Corrected Jul 17, 2026: we previously printed Anthropic's own \"~86%\" and claimed independent evals tracked it within ~1 point, which was wrong — the independent number is 2.6 points higher, and our methodology is to prefer the independent one. Run through the Claude Code harness instead of the bare bash agent, vals.ai measures 85.8%, a reminder that the harness moves these numbers as much as the model does. SWE-bench Pro 69.2% is Anthropic-reported.",
      "sameAs": [
        "https://www.anthropic.com/claude"
      ]
    },
    {
      "rank": 12,
      "name": "Grok 4.5",
      "maker": "SpaceXAI (xAI)",
      "open": false,
      "sweBenchVerified": "86.6%",
      "sweBenchVerifiedVendor": null,
      "sweBenchVerifiedIndependent": "86.6%",
      "independentSource": "vals.ai",
      "independentDate": "2026-07-14",
      "rankedOn": "independent",
      "sweBenchPro": "64.7%",
      "terminalBench": "83.3% (TB2.1)",
      "inputPricePer1M": "$2",
      "outputPricePer1M": "$6",
      "context": null,
      "verified": true,
      "announced": null,
      "source": {
        "title": "vals.ai — SWE-bench Verified (independent)",
        "url": "https://www.vals.ai/benchmarks/swebench"
      },
      "note": "Independent (vals.ai, Jul 14 2026, mini-swe-agent bash-only harness): SWE-bench Verified 86.6% ±1.52. Verified Jul 17, 2026 — it launched Jul 8 with no Verified score and we had it unranked as \"Opus-class, unproven\"; the independent number now backs the Opus-class claim, landing it 2 points under Claude Opus 4.8 at 40% of the input price. On our price-per-solved-task metric that is ~$2.31 against $5.64 for Opus 4.8 and $10.53 for Fable 5, the cheapest of any model scoring above 85%. It is also quick: 199.6s mean latency per task versus 566.9s for Opus 4.8. SWE-bench Pro 64.7% and Terminal-Bench 2.1 83.3% remain SpaceXAI-reported. Priced $2/$6 per 1M. Still not available in the EU.",
      "sameAs": []
    },
    {
      "rank": 13,
      "name": "Muse Spark 1.2",
      "maker": "Meta",
      "open": false,
      "sweBenchVerified": "86.6%",
      "sweBenchVerifiedVendor": null,
      "sweBenchVerifiedIndependent": "86.6%",
      "independentSource": "vals.ai",
      "independentDate": "2026-08-06",
      "rankedOn": "independent",
      "sweBenchPro": null,
      "terminalBench": "82.9% (TB2.1)",
      "inputPricePer1M": "$1.25",
      "outputPricePer1M": "$4.25",
      "context": "1M",
      "verified": true,
      "announced": null,
      "source": {
        "title": "vals.ai — SWE-bench Verified (independent)",
        "url": "https://www.vals.ai/benchmarks/swebench"
      },
      "note": "Independent (vals.ai, Aug 6 2026, mini-swe-agent bash-only harness): SWE-bench Verified 86.6%. Added Aug 7, 2026, the day after Meta launched it alongside the Muse Code terminal agent. This is an exact tie with Grok 4.5 at 86.6%, not a win over it; Grok 4.5 takes the higher rank only on the SWE-bench Pro tiebreak, since Meta has published no Pro figure. Meta published no SWE-bench Verified number of its own, so the independent score is the ranked one. A 4.6-point gain over Muse Spark 1.1 (82.0%) in four weeks. The weakness is long work: 64% on vals.ai's 1-to-4-hour task tier against 90% for Claude Opus 5 and 74% for Claude Opus 4.8, while short tasks reach 92%. Terminal-Bench 2.1 82.9% and DeepSWE 1.1 59.3% are Meta-reported, and Meta published them alongside Claude Opus 5's higher 86.7% and 65.0%. Standard pricing $1.25/$4.25 per 1M matches Muse Spark 1.1; a contributor tier cuts that to $0.10/$0.20 in exchange for permission to train on your prompts and completions. See /p/meta-muse-code-contributor-tier-data-pricing/.",
      "sameAs": []
    },
    {
      "rank": 14,
      "name": "Qwen3.8-27B",
      "maker": "Alibaba",
      "open": true,
      "sweBenchVerified": "86.0%",
      "sweBenchVerifiedVendor": null,
      "sweBenchVerifiedIndependent": "86.0%",
      "independentSource": "vals.ai",
      "independentDate": "2026-08-20",
      "rankedOn": "independent",
      "sweBenchPro": "61.7%",
      "terminalBench": "73.0 (TB2.1)",
      "inputPricePer1M": null,
      "outputPricePer1M": null,
      "context": "262K (1M extended)",
      "verified": true,
      "announced": null,
      "source": {
        "title": "vals.ai — SWE-bench Verified (independent)",
        "url": "https://www.vals.ai/benchmarks/swebench"
      },
      "note": "Independent (vals.ai, 2026-08-20 sweep, mini-swe-agent bash-only harness): SWE-bench Verified 86.0% ±1.55, 14th of 86 systems. Left the verifying queue on 2026-08-20, six days after launch. This is now the highest-scoring model on the board carrying an OSI-approved licence (Apache 2.0, weights on Hugging Face), ahead of GLM 5.2 at an independent 82.8%. Kimi K3 scores higher at 93.4% but ships under a bespoke non-OSI licence in 4-bit only, so K3 is open for inference where this is open outright. Notable for a dense 27B: it lands within error bars of Alibaba’s own 2.4T-parameter Qwen3.8-Max at 85.6%, meaning the flagship buys almost nothing on this benchmark. Vendor figures from the Aug 14 2026 model card, unchanged and still unconfirmed: SWE-bench Pro 61.7%, Terminal-Bench 2.1 73.0, GPQA Diamond 89.2, LiveCodeBench v6 90.3, plus vision scores (OSWorld-Verified 84.3, AndroidWorld 81.9). Alibaba published no SWE-bench Verified figure, so this rank rests entirely on the independent run. Weigh that against this maker’s record: Qwen3.7 Max carries the board’s widest vendor-versus-independent gap, an 80.4% claim against vals.ai’s 68.8%. See /p/qwen3-8-27b-open-weights-scoreboard-attached/.",
      "sameAs": []
    },
    {
      "rank": 15,
      "name": "Qwen3.8-Max",
      "maker": "Alibaba",
      "open": false,
      "sweBenchVerified": "85.6%",
      "sweBenchVerifiedVendor": null,
      "sweBenchVerifiedIndependent": "85.6%",
      "independentSource": "vals.ai",
      "independentDate": "2026-08-08",
      "rankedOn": "independent",
      "sweBenchPro": null,
      "terminalBench": null,
      "inputPricePer1M": "$2.00",
      "outputPricePer1M": "$6.00",
      "context": "1M",
      "verified": true,
      "announced": null,
      "source": {
        "title": "vals.ai — SWE-bench Verified (independent)",
        "url": "https://www.vals.ai/benchmarks/swebench"
      },
      "note": "Independent (vals.ai, benchmark updated 2026-08-08, mini-swe-agent bash-only harness): SWE-bench Verified 85.6% ± 1.57. Announced Aug 3, 2026 as \"a new bar for coding and cowork\" at 2.4T parameters. Alibaba published NO benchmark table with the launch: no SWE-bench Verified, no SWE-bench Pro, no Terminal-Bench, no methodology, so it sat unranked here from Aug 3 until this independent score appeared. That makes it one of the few rows on this board whose only number is independent, with no vendor claim to disclose against. Two caveats worth reading with the rank. It is effectively tied with Muse Spark 1.2 and Grok 4.5 above it (86.6% each; pooled SEM on a 1.0-point gap is about 2.2, so the gap is not significant) and with Claude Sonnet 5 below. And it is by far the slowest ranked model vals.ai measured, 2502s per test against 199s for Grok 4.5, roughly 12x, at $1.13 per test. For calibration, Qwen3.7 Max carries this board's widest vendor-versus-independent gap, an 80.4% claim against 68.8% measured. Price: $2.00/$6.00 per 1M, implicit caching $0.25/1M. See /p/qwen3-8-max-2-4t-launch-no-benchmarks/.",
      "sameAs": []
    },
    {
      "rank": 16,
      "name": "Claude Sonnet 5",
      "maker": "Anthropic",
      "open": false,
      "sweBenchVerified": "85.2%",
      "sweBenchVerifiedVendor": "85.2%",
      "sweBenchVerifiedIndependent": "79.6%",
      "independentSource": "vals.ai",
      "independentDate": "2026-08-13",
      "rankedOn": "vendor",
      "sweBenchPro": "63.2%",
      "terminalBench": "80.4% (TB2.1)",
      "inputPricePer1M": "$2",
      "outputPricePer1M": "$10",
      "context": "1M",
      "verified": true,
      "announced": null,
      "source": {
        "title": "GENZ TECH — Claude Sonnet 5, decoded",
        "url": "https://genztech.blog/p/claude-sonnet-5-near-opus-half-price/"
      },
      "note": "Vendor-reported (Anthropic), on Anthropic's own scaffold. Independent comparison: vals.ai's bash-only harness measures Sonnet 5 at 79.6% ±1.80, 5.6 points lower — a gap that reflects the harness as much as the model, so treat the 85.2% as a best-case number. Re-confirmed 2026-08-13: vals.ai still carries Sonnet 5 at 79.60% ±1.80, unchanged, so the 5.6-point vendor-versus-independent gap stands. Per this board's standing rule we keep the maker's figure as the ranked score and disclose the independent one here rather than rebasing. Intro pricing $2/$10 per 1M through Aug 31, 2026, then $3/$15.",
      "sameAs": [
        "https://www.anthropic.com/claude"
      ]
    },
    {
      "rank": 17,
      "name": "GLM 5.2",
      "maker": "Z.ai (Zhipu AI)",
      "open": true,
      "sweBenchVerified": "82.8%",
      "sweBenchVerifiedVendor": null,
      "sweBenchVerifiedIndependent": "82.8%",
      "independentSource": "vals.ai",
      "independentDate": "2026-07-22",
      "rankedOn": "independent",
      "sweBenchPro": null,
      "terminalBench": null,
      "inputPricePer1M": "$1.40",
      "outputPricePer1M": "$4.40",
      "context": "1M",
      "verified": true,
      "announced": null,
      "source": {
        "title": "vals.ai — SWE-bench Verified (independent)",
        "url": "https://www.vals.ai/benchmarks/swebench"
      },
      "note": "Independent (vals.ai, Jul 22 2026, mini-swe-agent bash-only harness): SWE-bench Verified 82.8% ±1.69, 9th of 75 systems. Added Jul 27, 2026 after a routine sweep found vals.ai had scored it and we were not carrying it. Z.ai published no SWE-bench Verified figure of its own, so the independent number is the ranked score with no vendor claim to disclose against it. This is now the highest-scoring genuinely downloadable model on the board: Kimi K3 scores higher at 93.4% but its weights remain unpublished, so K3 is listed as closed. Weights went up on Hugging Face and ModelScope under an unrestricted MIT license on Jun 16, 2026. Priced $1.40/$4.40 per 1M, cached input $0.26. vals.ai measured $0.71 per SWE-bench task, cheaper per solved task than every closed model ranked above it except GPT-5.6 Luna. Context 1M, max output 131k.",
      "sameAs": []
    },
    {
      "rank": 18,
      "name": "GPT-5.5",
      "maker": "OpenAI",
      "open": false,
      "sweBenchVerified": "82.6%",
      "sweBenchVerifiedVendor": null,
      "sweBenchVerifiedIndependent": "82.6%",
      "independentSource": "vals.ai",
      "independentDate": null,
      "rankedOn": "independent",
      "sweBenchPro": "58.6%",
      "terminalBench": "82.7% (TB2.0)",
      "inputPricePer1M": "$5",
      "outputPricePer1M": "$30",
      "context": null,
      "verified": true,
      "announced": null,
      "source": {
        "title": "vals.ai — SWE-bench Verified (independent)",
        "url": "https://www.vals.ai/benchmarks/swebench"
      },
      "note": "Verified score from vals.ai independent eval; Pro is OpenAI-reported (rivals flag possible memorization on Pro). Price: OpenAI list $5/$30 per 1M (cached input $0.50).",
      "sameAs": [
        "https://openai.com/"
      ]
    },
    {
      "rank": 19,
      "name": "Muse Spark 1.1",
      "maker": "Meta",
      "open": false,
      "sweBenchVerified": "82.0%",
      "sweBenchVerifiedVendor": null,
      "sweBenchVerifiedIndependent": "82.0%",
      "independentSource": "vals.ai",
      "independentDate": "2026-07-14",
      "rankedOn": "independent",
      "sweBenchPro": null,
      "terminalBench": null,
      "inputPricePer1M": "$1.25",
      "outputPricePer1M": "$4.25",
      "context": "1M",
      "verified": true,
      "announced": null,
      "source": {
        "title": "vals.ai — SWE-bench Verified (independent)",
        "url": "https://www.vals.ai/benchmarks/swebench"
      },
      "note": "Independent (vals.ai, Jul 14 2026, mini-swe-agent bash-only harness): SWE-bench Verified 82.0% ±1.72. Verified Jul 17, 2026; Meta published no SWE-bench number of its own and still has not. Do not confuse this with the earlier Muse Spark, which vals.ai scores 74.4% — 1.1 is a 7.6-point improvement over its predecessor. Price-per-solved-task ~$1.52, the lowest of any model in the top ten. Weakness is long tasks: 52% on the 1-4 hour tier against 74% for Claude Opus 4.8. Launched Jul 9 2026 by Meta Superintelligence Labs; $1.25/$4.25 per 1M with $20 starter credits.",
      "sameAs": []
    },
    {
      "rank": 20,
      "name": "Gemini 3.7 Flash",
      "maker": "Google DeepMind",
      "open": false,
      "sweBenchVerified": "80.8%",
      "sweBenchVerifiedVendor": null,
      "sweBenchVerifiedIndependent": "80.8%",
      "independentSource": "vals.ai",
      "independentDate": "2026-08-17",
      "rankedOn": "independent",
      "sweBenchPro": null,
      "terminalBench": null,
      "inputPricePer1M": "$0.75",
      "outputPricePer1M": "$3.75",
      "context": null,
      "verified": true,
      "announced": null,
      "source": {
        "title": "vals.ai — SWE-bench Verified (independent)",
        "url": "https://www.vals.ai/benchmarks/swebench"
      },
      "note": "Independent (vals.ai, mini-swe-agent bash-only harness): SWE-bench Verified 80.80% ±1.76. Released Aug 13, 2026, three weeks after Gemini 3.6 Flash. Google published NO SWE-bench Verified score of its own, exactly as with 3.6 Flash, so this rank rests entirely on the independent run. Its launch claims are all vendor-reported on Google scaffolding: DeepSWE v1.1 65.3% (up from 49.0%), FrontierCode 1.1 Main 43.6% (up from 34.4%), WebDev Arena Elo 1588 (up from 1538), GDP.pdf 34.0% (up from 22.0%), AutomationBench 30.4% (up from 17.0%). Read those with the 3.6 Flash precedent in mind: Google claimed a 12-point DeepSWE jump for that model too, and vals.ai then measured it at 79.60% ±1.80 against 78.8% for 3.5 Flash, a statistical tie. Price: introductory $0.75/$3.75 per 1M through Dec 31, 2026, reverting to $1.50/$7.50 on Jan 1, 2027, i.e. back to 3.6 Flash list. Specs: 1,048,576 input / 65,536 output tokens; thinking levels low/medium/high only (minimal now errors); computer use still preview. See /p/gemini-3-7-flash-half-price-3-5-pro-missing/. **Ranked 2026-08-17**: vals.ai evaluated it and we now rank on that independent figure, since Google still publishes no SWE-bench Verified score of its own. It entered our verifying queue on Aug 13 and left it four days later. Worth recording that the 3.6 Flash precedent did NOT repeat: that model turned a claimed 12-point DeepSWE jump into a statistical tie, whereas 3.7 Flash gains a real 1.2 points over 3.6 Flash (79.60%) on the same harness. The gain is still inside the combined margin of error, so read 3.7 Flash, 3.6 Flash and Claude Sonnet 5 as a three-way tie rather than a ranking.",
      "sameAs": []
    },
    {
      "rank": 21,
      "name": "DeepSeek V4 Pro",
      "maker": "DeepSeek",
      "open": true,
      "sweBenchVerified": "80.6%",
      "sweBenchVerifiedVendor": "80.6%",
      "sweBenchVerifiedIndependent": "96.4%",
      "independentSource": "vals.ai",
      "independentDate": "2026-08-13",
      "rankedOn": "vendor",
      "sweBenchPro": "55.4%",
      "terminalBench": "67.9% (TB2.0)",
      "inputPricePer1M": "$0.435",
      "outputPricePer1M": "$0.87",
      "context": "1M",
      "verified": true,
      "announced": null,
      "source": {
        "title": "DeepSeek-V4-Pro model card (vendor)",
        "url": "https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro"
      },
      "note": "Vendor-reported: DeepSeek's own model card, Pro-Max mode (SWE-bench Verified 80.6%, SWE-bench Pro 55.4%). Updated Aug 12, 2026: DeepSeek shipped the GA build, DeepSeek-V4-Pro-0813, and repointed the deepseek-v4-pro API alias to it. The published benchmark table did not change, so the ranked score is unmoved; the source link was upgraded from an llm-stats-derived aggregator page to DeepSeek's primary model card. Corrected Jul 17, 2026: we previously called llm-stats an \"independent tracker\", but llm-stats labels its own SWE-bench Verified table \"Verified: 0 / Self-reported: 104\", so every score on it is vendor-claimed rather than independent. **An independent score now exists, and it inverts this row, 2026-08-13.** vals.ai has evaluated DeepSeek V4 Pro 0813 at 96.40% ±0.83 on the mini-swe-agent bash-only harness, placing it 2nd of 82 systems, 0.6 points off Claude Opus 5. That is the same GA build this row already documents above (DeepSeek repointed the deepseek-v4-pro API alias to 0813 on Aug 12), so it is an exact match, not a near-miss. It is also 15.8 points ABOVE DeepSeek's own 80.6% claim, which is the opposite direction from every other disclosure row on this board, where vendors run 4.0 to 11.6 points high. We have NOT rebased the ranked score, because this board's standing decision is DISCLOSE, DON'T REBASE: where a vendor figure and an exact independent figure both exist, the maker's number stays ranked and the independent one is disclosed here. That rule was written for vendor overstatement and does not obviously fit a vendor understating its own model by 15.8 points, so this row is flagged for a human decision on whether to rebase it to 96.40% (which would move it to 2nd, behind only Claude Opus 5, and make it the highest-scoring open-weight model on the board). Until that call is made, the ranked score stays at the vendor's 80.6% and readers get the independent number here. vals.ai has also evaluated the plain DeepSeek V4 (77.4%), a different model, and DeepSeek's own smaller V4 Flash 0731 at 88.8%. Tied with Gemini 3.1 Pro on Verified, ahead on Pro. See /p/deepseek-v4-pro-ga-flash-still-scores-higher/.",
      "sameAs": [
        "https://www.deepseek.com/",
        "https://huggingface.co/deepseek-ai"
      ]
    },
    {
      "rank": 22,
      "name": "Gemini 3.1 Pro",
      "maker": "Google DeepMind",
      "open": false,
      "sweBenchVerified": "80.6%",
      "sweBenchVerifiedVendor": "80.6%",
      "sweBenchVerifiedIndependent": null,
      "independentSource": null,
      "independentDate": null,
      "rankedOn": "vendor",
      "sweBenchPro": "54.2%",
      "terminalBench": null,
      "inputPricePer1M": "$2",
      "outputPricePer1M": "$12",
      "context": null,
      "verified": true,
      "announced": null,
      "source": {
        "title": "Google DeepMind — Gemini Pro",
        "url": "https://deepmind.google/models/gemini/pro/"
      },
      "note": "Vendor-reported (DeepMind) pass rate. No independent eval of this exact model; vals.ai has run Gemini 3.1 Pro Preview (02/26) at 78.8%, a preview build we do not treat as the same model. Ties DeepSeek V4 Pro on Verified, trails it on Pro. Price: Google list $2/$12 per 1M for context up to 200K (doubles above 200K).",
      "sameAs": [
        "https://deepmind.google/models/gemini/"
      ]
    },
    {
      "rank": 23,
      "name": "MiniMax M3",
      "maker": "MiniMax",
      "open": true,
      "sweBenchVerified": "80.5%",
      "sweBenchVerifiedVendor": "80.5%",
      "sweBenchVerifiedIndependent": "75.0%",
      "independentSource": "vals.ai",
      "independentDate": null,
      "rankedOn": "vendor",
      "sweBenchPro": "59.0%",
      "terminalBench": "66.0 (TB2.1)",
      "inputPricePer1M": "$0.60",
      "outputPricePer1M": null,
      "context": "1M",
      "verified": true,
      "announced": null,
      "source": {
        "title": "VentureBeat — MiniMax-M3 debut",
        "url": "https://venturebeat.com/technology/minimax-m3-debuts-eclipsing-gpt-5-5-and-gemini-3-1-pro-on-key-benchmark-performance-for-just-5-10-of-the-cost"
      },
      "note": "Vendor-reported at launch (Jun 1, 2026). Independent comparison: vals.ai's bash-only harness measures MiniMax-M3 at 75.0% ±1.94, 5.5 points lower, so treat the 80.5% as a best-case number.",
      "sameAs": [
        "https://www.minimax.io/"
      ]
    },
    {
      "rank": 24,
      "name": "Qwen3.7 Max",
      "maker": "Alibaba",
      "open": false,
      "sweBenchVerified": "80.4%",
      "sweBenchVerifiedVendor": "80.4%",
      "sweBenchVerifiedIndependent": "68.8%",
      "independentSource": "vals.ai",
      "independentDate": null,
      "rankedOn": "vendor",
      "sweBenchPro": "60.6%",
      "terminalBench": "69.7 (TB2.0)",
      "inputPricePer1M": "$1.25",
      "outputPricePer1M": "$3.75",
      "context": "1M",
      "verified": true,
      "announced": null,
      "source": {
        "title": "W&B ml-news — Qwen3.7-Max benchmark scores",
        "url": "https://wandb.ai/byyoung3/ml-news/reports/Qwen3-7-Max-Benchmark-Scores---VmlldzoxNjk1MzA1MQ"
      },
      "note": "Vendor-reported (May 20, 2026), and the widest vendor-versus-independent gap on this board: vals.ai's bash-only harness measures Qwen 3.7 Max at 68.8% ±2.07, 11.6 points below Alibaba's claim — on that harness it would rank far down the table rather than here. Read the 80.4% as a best-case, own-scaffold number and weight it accordingly. Proprietary — the open-weights sibling is Qwen3.6-35B. Price: Alibaba Cloud (Singapore) promo $1.25/$3.75 per 1M; list $2.50/$7.50.",
      "sameAs": [
        "https://qwenlm.github.io/"
      ]
    },
    {
      "rank": 25,
      "name": "Kimi K2.6",
      "maker": "Moonshot AI",
      "open": true,
      "sweBenchVerified": "80.2%",
      "sweBenchVerifiedVendor": "80.2%",
      "sweBenchVerifiedIndependent": "76.2%",
      "independentSource": "vals.ai",
      "independentDate": null,
      "rankedOn": "vendor",
      "sweBenchPro": "58.6%",
      "terminalBench": null,
      "inputPricePer1M": "$0.95",
      "outputPricePer1M": "$4",
      "context": null,
      "verified": true,
      "announced": null,
      "source": {
        "title": "Moonshot AI — Kimi K2.6 (Hugging Face)",
        "url": "https://huggingface.co/moonshotai/Kimi-K2.6"
      },
      "note": "Vendor-reported (10-run average on Moonshot's own SWE-agent harness). Independent comparison: vals.ai's bash-only harness measures Kimi K2.6 at 76.2% ±1.91, 4 points lower. Price: official Moonshot API $0.95/$4 per 1M (cache-hit input $0.16).",
      "sameAs": [
        "https://huggingface.co/moonshotai"
      ]
    },
    {
      "rank": 26,
      "name": "Gemini 3.8 Flash",
      "maker": "Google DeepMind",
      "open": false,
      "sweBenchVerified": "80.0%",
      "sweBenchVerifiedVendor": null,
      "sweBenchVerifiedIndependent": "80.0%",
      "independentSource": "vals.ai",
      "independentDate": "2026-09-03",
      "rankedOn": "independent",
      "sweBenchPro": null,
      "terminalBench": "90.8%",
      "inputPricePer1M": "$0.75",
      "outputPricePer1M": "$3.75",
      "context": null,
      "verified": true,
      "announced": null,
      "source": {
        "title": "vals.ai — SWE-bench Verified (independent)",
        "url": "https://www.vals.ai/benchmarks/swebench"
      },
      "note": "Independent (vals.ai, observed 2026-09-03, mini-swe-agent bash-only harness): SWE-bench Verified 80.00% ±1.79. Leaves the verifying queue it entered on 2026-09-02, when a model card briefly leaked at deepmind.google/models/model-cards/gemini-3-8-flash/ before being pulled, and the Wall Street Journal reported Google was preparing to ship it under the internal codename \"Skimaki.\" Google officially launched the model Sept 2, 2026 at $0.75/$3.75 per 1M tokens through Dec 31, 2026 (reverting to $1.50/$7.50 on Jan 1, 2027, matching the 3.7 Flash pricing pattern). Google's own claimed Terminal-Bench 2.1 score is 90.8%, run on Google's own scaffolding and not independently reproduced here. Ranks 26th on the independent SWE-bench Verified figure (25th when it entered on 2026-09-03; Ember-1's arrival on 2026-09-28 moved it down one), essentially tied with Kimi K2.6 (80.2%) just above and Gemini 3.6 Flash (79.6%) just below, both within the combined margin of error, so read this as a three-way statistical tie rather than a clean ordering. Google has published no SWE-bench Verified number of its own for direct comparison.",
      "sameAs": []
    },
    {
      "rank": 27,
      "name": "Gemini 3.6 Flash",
      "maker": "Google DeepMind",
      "open": false,
      "sweBenchVerified": "79.6%",
      "sweBenchVerifiedVendor": null,
      "sweBenchVerifiedIndependent": "79.6%",
      "independentSource": "vals.ai",
      "independentDate": "2026-07-23",
      "rankedOn": "independent",
      "sweBenchPro": null,
      "terminalBench": null,
      "inputPricePer1M": "$1.50",
      "outputPricePer1M": "$7.50",
      "context": null,
      "verified": true,
      "announced": null,
      "source": {
        "title": "vals.ai — SWE-bench Verified (independent)",
        "url": "https://www.vals.ai/benchmarks/swebench"
      },
      "note": "Independent (vals.ai, mini-swe-agent bash-only harness): SWE-bench Verified 79.60% ±1.80. Verified Jul 23, 2026 — vals.ai had not run it at our Jul 22 check, two days after launch, and it enters the board at 15th, 0.8 points above the older Gemini 3.5 Flash (78.8%). That gap is well inside the combined error bars (~0.3 sigma), so read the two Flash tiers as a statistical tie on coding, not an upgrade. Google published no SWE-bench Verified score of its own; its launch claims were DeepSWE 49% (up from 37%) and OSWorld-Verified 83.0% (up from 78.4%), which we could not rank on. Price: Google list $1.50/$7.50 per 1M, cheaper on output than 3.5 Flash at $9.",
      "sameAs": []
    },
    {
      "rank": 28,
      "name": "Composer 2.5",
      "maker": "Cursor",
      "open": false,
      "sweBenchVerified": "79.6%",
      "sweBenchVerifiedVendor": null,
      "sweBenchVerifiedIndependent": "79.6%",
      "independentSource": "vals.ai",
      "independentDate": "2026-07-22",
      "rankedOn": "independent",
      "sweBenchPro": null,
      "terminalBench": null,
      "inputPricePer1M": "$0.50",
      "outputPricePer1M": "$2.50",
      "context": "200k",
      "verified": true,
      "announced": null,
      "source": {
        "title": "vals.ai — SWE-bench Verified (independent)",
        "url": "https://www.vals.ai/benchmarks/swebench"
      },
      "note": "Independent (vals.ai, Jul 22 2026, mini-swe-agent bash-only harness): SWE-bench Verified 79.6% ±1.80, 13th of 75 systems. Added Jul 27, 2026 from a routine sweep. Statistically tied with Gemini 3.6 Flash, also 79.6%: the pooled standard error is about 2.5 points, so treat the two as level and the ordering between them as arbitrary. Cursor reports the model is built on Moonshot's open-weight Kimi K2.5 base with roughly 85% of total compute from Cursor's own post-training, but Composer 2.5 itself is not released, so it is listed as closed. Priced $0.50/$2.50 per 1M inside Cursor, with a faster variant at $3.00/$15.00. Context 200k. Released May 18, 2026.",
      "sameAs": []
    },
    {
      "rank": 29,
      "name": "Gemini 3.5 Flash",
      "maker": "Google DeepMind",
      "open": false,
      "sweBenchVerified": "78.8%",
      "sweBenchVerifiedVendor": null,
      "sweBenchVerifiedIndependent": "78.8%",
      "independentSource": "vals.ai",
      "independentDate": null,
      "rankedOn": "independent",
      "sweBenchPro": null,
      "terminalBench": null,
      "inputPricePer1M": "$1.50",
      "outputPricePer1M": "$9",
      "context": null,
      "verified": true,
      "announced": null,
      "source": {
        "title": "vals.ai — SWE-bench Verified (independent)",
        "url": "https://www.vals.ai/benchmarks/swebench"
      },
      "note": "vals.ai independent eval. See our decode of its native computer-use tool. Price: Google list $1.50/$9 per 1M (cached input $0.15).",
      "sameAs": [
        "https://deepmind.google/models/gemini/"
      ]
    },
    {
      "rank": 30,
      "name": "Gemini 3.5 Flash-Lite",
      "maker": "Google DeepMind",
      "open": false,
      "sweBenchVerified": "75.0%",
      "sweBenchVerifiedVendor": null,
      "sweBenchVerifiedIndependent": "75.0%",
      "independentSource": "vals.ai",
      "independentDate": "2026-07-23",
      "rankedOn": "independent",
      "sweBenchPro": "54.2%",
      "terminalBench": "54%",
      "inputPricePer1M": "$0.30",
      "outputPricePer1M": "$2.50",
      "context": null,
      "verified": true,
      "announced": null,
      "source": {
        "title": "vals.ai — SWE-bench Verified (independent)",
        "url": "https://www.vals.ai/benchmarks/swebench"
      },
      "note": "Independent (vals.ai, mini-swe-agent bash-only harness): SWE-bench Verified 75.00% ±1.94. Verified Jul 23, 2026 — vals.ai had not run it at our Jul 22 check and it now enters the board last at 17th, but it is the cost outlier: at $0.30/$2.50 per 1M it lands within 3.8 points of Gemini 3.5 Flash (78.8%) for a fifth of the input price, and that gap is inside the combined error bars. SWE-bench Pro 54.2% and Terminal-Bench 2.1 54% remain vendor-reported (Google, Jul 21 2026). Price: Google list $0.30/$2.50 per 1M.",
      "sameAs": []
    },
    {
      "rank": null,
      "name": "GPT-6.1 Sol",
      "maker": "OpenAI",
      "open": false,
      "sweBenchVerified": null,
      "sweBenchVerifiedVendor": null,
      "sweBenchVerifiedIndependent": null,
      "independentSource": null,
      "independentDate": null,
      "rankedOn": null,
      "sweBenchPro": null,
      "terminalBench": null,
      "inputPricePer1M": "$2",
      "outputPricePer1M": "$10",
      "context": null,
      "verified": false,
      "announced": "2026-09-29",
      "source": {
        "title": "GENZ TECH — GPT-6.1 Sol: OpenAI's cheap model nears Astra's power",
        "url": "https://genztech.blog/p/gpt-6-1-sol-openai-cheap-model-nears-astra/"
      },
      "note": "Vendor-reported (OpenAI, Sept 29 2026): matches GPT-6 Astra on DeepSWE v1.1 (a real-codebase coding benchmark) at roughly a fifth of the cost, and scores 2.2 points above Claude Opus 5.5 on AutomationBench at medium reasoning effort. OpenAI did not publish a SWE-bench Verified score for this release, so there is nothing to rank on our tracked metric; it enters the verifying queue unranked. Pricing: $2/M input, $10/M output, $0.10/M cached input, versus Astra's $10/$50, matching Claude Sonnet 5's list price to the cent. Queue caveat: vals.ai archived its SWE-bench Verified board on 5 September 2026, so this row is unlikely to ever receive an independent score from that source. See /p/gpt-6-1-sol-openai-cheap-model-nears-astra/. Re-checked 2026-10-16: vals.ai has not evaluated it; its SWE-bench Verified board is archived (88 systems, frozen since 2026-09-05).",
      "sameAs": []
    },
    {
      "rank": null,
      "name": "Claude Sonnet 5.5",
      "maker": "Anthropic",
      "open": false,
      "sweBenchVerified": null,
      "sweBenchVerifiedVendor": null,
      "sweBenchVerifiedIndependent": null,
      "independentSource": null,
      "independentDate": null,
      "rankedOn": null,
      "sweBenchPro": null,
      "terminalBench": "70.6 (TB4, vendor)",
      "inputPricePer1M": "$2",
      "outputPricePer1M": "$10",
      "context": null,
      "verified": false,
      "announced": "2026-09-28",
      "source": {
        "title": "GENZ TECH — Anthropic launches Claude Sonnet 5.5",
        "url": "https://genztech.blog/p/claude-sonnet-5-5-launch/"
      },
      "note": "Vendor-reported (Anthropic, Sept 28 2026): Terminal-Bench 4.0 70.6%, FrontierCode 1.1 46.2% (max effort), CursorBench 4.0 55.5%, GDPval-AA v2.1 1844 Elo, OSWorld 2.1 80.1% (partial credit). Anthropic published NO SWE-bench Verified score for this model, continuing the pivot away from that metric it started with Opus 5.5. List pricing is unchanged from Sonnet 5 at $2/$10 per 1M tokens; Anthropic's '30% cheaper per task' claim is a total-cost-per-completed-job figure driven by a stated 30%+ speed gain and fewer output tokens, not a rate cut. Queue caveat: vals.ai archived its SWE-bench Verified board on 5 September 2026, so this row is unlikely to ever receive an independent score from that source. See /p/claude-sonnet-5-5-launch/. Re-checked 2026-10-16: vals.ai has not evaluated it; its SWE-bench Verified board is archived (88 systems, frozen since 2026-09-05).",
      "sameAs": []
    },
    {
      "rank": null,
      "name": "Claude Opus 5.5",
      "maker": "Anthropic",
      "open": false,
      "sweBenchVerified": null,
      "sweBenchVerifiedVendor": null,
      "sweBenchVerifiedIndependent": null,
      "independentSource": null,
      "independentDate": null,
      "rankedOn": null,
      "sweBenchPro": null,
      "terminalBench": "66.4 (TB4, vendor)",
      "inputPricePer1M": "$4",
      "outputPricePer1M": "$20",
      "context": null,
      "verified": false,
      "announced": "2026-09-22",
      "source": {
        "title": "GENZ TECH — Claude Opus 5.5 and the shift away from SWE-bench",
        "url": "https://genztech.blog/p/claude-opus-5-5-terminal-bench-agentic-benchmarks/"
      },
      "note": "Vendor-reported (Anthropic, Sept 22 2026): Terminal-Bench 4.0 66.4%, FrontierCode v1.1 54.4%, CursorBench 4.0 57.8%, GDPval-AA v2.1 1846 Elo, AutomationBench 40.0%, OSWorld 2.0 81.8%, Chartography 89.0%. Anthropic published NO SWE-bench Verified score for this model, so there is nothing to rank on our tracked metric and it enters the verifying queue unranked. Pricing precision: list price is $4/$20 per 1M against Opus 5's $5/$25, i.e. 20% lower per token; Anthropic's own claim of '40% less to run than Opus 5' is a total-cost claim that also folds in a stated 30% faster output, not a per-token discount, so the two figures are not in conflict but are not the same measure either. Queue caveat: vals.ai archived its SWE-bench Verified board on 5 September 2026, so this row is unlikely to ever receive an independent score from that source. See /p/claude-opus-5-5-terminal-bench-agentic-benchmarks/. Re-checked 2026-10-16: vals.ai has not evaluated it; its SWE-bench Verified board is archived (88 systems, frozen since 2026-09-05) and the nearest names on it are Claude Opus 5 and Claude Opus 4.8, neither of which is this model.",
      "sameAs": []
    },
    {
      "rank": null,
      "name": "Grok 4.7",
      "maker": "SpaceXAI (xAI)",
      "open": false,
      "sweBenchVerified": null,
      "sweBenchVerifiedVendor": null,
      "sweBenchVerifiedIndependent": null,
      "independentSource": null,
      "independentDate": null,
      "rankedOn": null,
      "sweBenchPro": null,
      "terminalBench": null,
      "inputPricePer1M": "$2",
      "outputPricePer1M": "$6",
      "context": null,
      "verified": false,
      "announced": "2026-09-21",
      "source": {
        "title": "SpaceXAI — Introducing Grok 4.7",
        "url": "https://x.ai/news/grok-4-7"
      },
      "note": "Released Sep 21, 2026, about nine days after Musk teased a Sep 11-12 target. SpaceXAI describes a new, larger base model with extended RL training for complex, multi-hour tasks and native Grok Bot harness integration, but discloses no parameter count. Vendor-reported only, on benchmarks SpaceXAI controls: CursorBench v4.0 46.3% (leads price-performance; NOT comparable to Grok 4.6's 69.9% on the older v3.2), plus three professional-domain evals with no rival baseline: electrical engineering 64%, clinical reasoning 56.7%, legal work 19.6%. No SWE-bench Verified or SWE-bench Pro figure published. Pricing unchanged from Grok 4.6 at $2/$6 per 1M. Complicating factor: vals.ai archived its SWE-bench Verified board on Sep 5, 2026 citing score saturation, so an independent score on our tracked metric may not arrive at all. Enters the verifying queue unranked. See /p/grok-4-7-spacexai-coding-model-delay/. Re-checked 2026-10-16: vals.ai has not evaluated it; its SWE-bench Verified board is archived (88 systems, frozen since 2026-09-05). It lists Grok 4.6 (95.6%) and Grok 4.5 (86.6%), neither of which is this model.",
      "sameAs": []
    },
    {
      "rank": null,
      "name": "Step 5 Preview",
      "maker": "StepFun",
      "open": false,
      "sweBenchVerified": null,
      "sweBenchVerifiedVendor": null,
      "sweBenchVerifiedIndependent": null,
      "independentSource": null,
      "independentDate": null,
      "rankedOn": null,
      "sweBenchPro": null,
      "terminalBench": "33.3 (TB4, vendor)",
      "inputPricePer1M": "$1.00",
      "outputPricePer1M": "$2.70",
      "context": "1M",
      "verified": false,
      "announced": "2026-09-20",
      "source": {
        "title": "GENZ TECH — StepFun Step 5 Preview: 600B agent model at $1 per million tokens",
        "url": "https://genztech.blog/p/stepfun-step-5-preview-600b-moe-1-dollar-per-million-tokens/"
      },
      "note": "Vendor-reported (StepFun, Sept 20 2026): Terminal-Bench v4 33.3% (vs Claude Opus 5 52.3%, GPT-6 Astra 57.9%), DeepSWE v1.1 67.7%, ProgramBench 80.5%, StepCodeBench 49, Agents' Last Exam 29.5%, FrontierFinance 66.4%, DRACO 83.3%. StepFun published NO SWE-bench Verified score and no SWE-bench Pro, so there is nothing to rank on our tracked metric; it enters the verifying queue unranked. Independent context, not a SWE-bench number: Artificial Analysis scores it 44 on its Intelligence Index (27th of 653), tied with Kimi K3 (max) at roughly 65% lower cost per task, at about 99.8 output tokens/s. Price confirmed via Artificial Analysis: $1.00/$2.70 per 1M, cached input about $0.05. Re-checked 2026-10-16: vals.ai's SWE-bench Verified board remains archived (frozen for posterity, no new models are being scored; 88 systems listed) and shows no evaluation for this model. See /p/stepfun-step-5-preview-600b-moe-1-dollar-per-million-tokens/.",
      "sameAs": []
    },
    {
      "rank": null,
      "name": "Cognition SWE-2",
      "maker": "Cognition",
      "open": false,
      "sweBenchVerified": null,
      "sweBenchVerifiedVendor": null,
      "sweBenchVerifiedIndependent": null,
      "independentSource": null,
      "independentDate": null,
      "rankedOn": null,
      "sweBenchPro": null,
      "terminalBench": null,
      "inputPricePer1M": null,
      "outputPricePer1M": null,
      "context": null,
      "verified": false,
      "announced": "2026-09-10",
      "source": {
        "title": "Cognition — Introducing SWE-2",
        "url": "https://cognition.com/blog/swe-2"
      },
      "note": "Released Sept 10, 2026. Cognition's own numbers, all on benchmarks Cognition built or controls (FrontierCode 1.1 Main, DeepSWE 1.1, Terminal-Bench 2.1/4), not SWE-bench Verified: FrontierCode 1.1 Main 50.0% (vs SWE-1.7's 42.0%, Fable 5.1's 50.9%, GPT-6 Astra's 53.3%), DeepSWE 1.1 73.0%, Terminal-Bench 2.1 92.8%. Cognition claims an 81% average cost cut vs SWE-1.7 (58% fewer steps: 53 vs 127 on medium effort) and a 64% cost cut vs Fable 5.1 for a score within one point of it. No SWE-bench Verified score published by Cognition or measured independently, and no token pricing or context window disclosed. Complicating factor: vals.ai archived its SWE-bench Verified board on Sept 5, 2026 citing score saturation, so this model may not get an independent score on our tracked metric at all. Enters the verifying queue unranked. Re-checked 2026-10-16: vals.ai's board remains archived (no new models being scored, 88 systems listed) and still shows no evaluation for this model. See /p/cognition-swe-2-coding-model-launch/.",
      "sameAs": []
    },
    {
      "rank": null,
      "name": "Muse Glimmer",
      "maker": "Meta",
      "open": true,
      "sweBenchVerified": null,
      "sweBenchVerifiedVendor": null,
      "sweBenchVerifiedIndependent": null,
      "independentSource": null,
      "independentDate": null,
      "rankedOn": null,
      "sweBenchPro": null,
      "terminalBench": null,
      "inputPricePer1M": "Free",
      "outputPricePer1M": "Free",
      "context": null,
      "verified": false,
      "announced": "2026-08-10",
      "source": {
        "title": "Meta AI Research — Introducing Muse Glimmer",
        "url": "https://research.meta.ai/blog/introducing-muse-glimmer-open-agentic-model"
      },
      "note": "Released Aug 10, 2026 with weights on Hugging Face under a plain Apache 2.0 license, Meta's first genuinely permissive open-weight release in the Muse line, which had been API-only until now. Meta published NO SWE-bench Verified score, no SWE-bench Pro and no Terminal-Bench figure: the launch post benchmarks it against Gemma4-31B and Qwen3.6-27B in its own size class and says only that it \"performs strongly for its size class\". With no number of any provenance, vendor or independent, there is nothing to rank, so it enters unranked. vals.ai has not evaluated it. Specs that are confirmed: 30B dense, logit-distilled from Muse Spark as teacher, multimodal via a dedicated perception encoder, 100+ languages, OpenClaw scaffold compatible. Memory: 55GB+ at full bf16, compressed to under 20GB by Meta's K-Quant-17GB build (roughly 4-bit), which is what lets the weights, KV cache, perception encoder and speculative-decoding drafter share a 24GB or 32GB card. Meta reports DFlash speculative decoding at 3.1x on an RTX 5090, 1.8x on an M5 Max and 1.5x on an M4 Max, all vendor-reported. Separately, Mark Zuckerberg and Alexandr Wang both said an open-weight version of Muse Spark 1.2 (ranked #9 here at 86.6%) follows in the coming weeks; the license for that release has not been stated, and it is a roadmap item until the files land. See /p/meta-muse-glimmer-open-weights-30b/. Re-checked 2026-10-16: vals.ai's SWE-bench Verified board remains archived (frozen for posterity, no new models are being scored; 88 systems listed) and still shows no evaluation for this model; all 22 previously-tracked ranked scores were re-verified this run against the frozen table and are unchanged, exact match.",
      "sameAs": []
    },
    {
      "rank": null,
      "name": "Laguna S 2.1",
      "maker": "Poolside",
      "open": true,
      "sweBenchVerified": null,
      "sweBenchVerifiedVendor": null,
      "sweBenchVerifiedIndependent": null,
      "independentSource": null,
      "independentDate": null,
      "rankedOn": null,
      "sweBenchPro": "59.4%",
      "terminalBench": "70.2% (TB2.1)",
      "inputPricePer1M": "$0.10",
      "outputPricePer1M": "$0.20",
      "context": "1M",
      "verified": false,
      "announced": "2026-07-21",
      "source": {
        "title": "GENZ TECH — Poolside Laguna S 2.1",
        "url": "https://genztech.blog/p/poolside-laguna-s-2-1-open-weights-8b-active/"
      },
      "note": "Vendor-reported (Poolside, Jul 21 2026): Terminal-Bench 2.1 70.2% with thinking enabled (60.4% without), SWE-bench Pro 59.4%, SWE-bench Multilingual 78.5%, DeepSWE v1.1 40.4%. Poolside published no SWE-bench Verified score, so it stays unranked pending an independent eval. vals.ai has evaluated Laguna M.1 (57.6%) and Laguna XS.2 (55.2%) but not S 2.1, and a near-miss version is not a match. Price: OpenRouter $0.10/$0.20 per 1M. Context added 2026-08-23: Nvidia is paying Poolside $6B for a non-exclusive license to the Model Factory system that builds this family, is investing $1B at a $13B post-money valuation, and has made job offers to the 109 engineers who built Laguna, per Bloomberg. Poolside stays independent and keeps shipping, so the row is unaffected, but who is training future Laguna builds is now an open question. See /p/nvidia-poolside-6b-license-model-factory-109-staff/. Re-checked 2026-10-16: vals.ai's SWE-bench Verified board remains archived (frozen for posterity, no new models are being scored; 88 systems listed) and still shows no evaluation for this model; all 22 previously-tracked ranked scores were re-verified this run against the frozen table and are unchanged, exact match.",
      "sameAs": []
    },
    {
      "rank": null,
      "name": "LongCat-2.0",
      "maker": "Meituan",
      "open": true,
      "sweBenchVerified": null,
      "sweBenchVerifiedVendor": null,
      "sweBenchVerifiedIndependent": null,
      "independentSource": null,
      "independentDate": null,
      "rankedOn": null,
      "sweBenchPro": "59.5%",
      "terminalBench": "70.8%",
      "inputPricePer1M": "$0.30",
      "outputPricePer1M": null,
      "context": "1M",
      "verified": false,
      "announced": "2026-06-30",
      "source": {
        "title": "GENZ TECH — Meituan LongCat-2.0",
        "url": "https://genztech.blog/p/meituan-longcat-2-open-source-coding-model/"
      },
      "note": "Vendor-reported (Meituan, Jun 30 2026): SWE-bench Pro 59.5%, Terminal-Bench 70.8%. No SWE-bench Verified score or independent eval published yet, so unranked pending confirmation. vals.ai has not evaluated it and does not cover Meituan at all, and llm-stats does not list the model. Re-checked 2026-10-16: vals.ai's SWE-bench Verified board remains archived (frozen for posterity, no new models are being scored; 88 systems listed) and still shows no evaluation for this model; all 22 previously-tracked ranked scores were re-verified this run against the frozen table and are unchanged, exact match.",
      "sameAs": []
    },
    {
      "rank": null,
      "name": "MiMo-V2.6-Pro",
      "maker": "Xiaomi",
      "open": true,
      "sweBenchVerified": null,
      "sweBenchVerifiedVendor": null,
      "sweBenchVerifiedIndependent": null,
      "independentSource": null,
      "independentDate": null,
      "rankedOn": null,
      "sweBenchPro": null,
      "terminalBench": null,
      "inputPricePer1M": "$0.43",
      "outputPricePer1M": "$0.87",
      "context": "1M",
      "verified": false,
      "announced": null,
      "source": {
        "title": "GENZ TECH — Xiaomi's MiMo-V2.6-Pro tops the open-weights rankings",
        "url": "https://genztech.blog/p/xiaomi-mimo-v2-6-pro-top-open-weights-model/"
      },
      "note": "No SWE-bench Verified score exists for this model from Xiaomi or any independent evaluator, so it enters the verifying queue unranked; we do not estimate. Independent context, not a SWE-bench number: Artificial Analysis scores MiMo-V2.6-Pro 46 on its Intelligence Index v4.3.2, the highest open-weights result on that board, ahead of GLM-5.3 (max) at 45 and Kimi K3 (max) at 44. That index is a composite of 10 evaluations including Terminal-Bench 4.0, GDPval-AA v2.1, AutomationBench-AA, SciCode and Humanity's Last Exam, so it is not comparable to the SWE-bench Verified figures this board ranks on. Architecture and price confirmed via Artificial Analysis: 1.0T total / 42B active MoE, 1M context, $0.43/$0.87 per 1M with a 99% cache discount, about 76.2 output tokens/s. Queue caveat: vals.ai archived its SWE-bench Verified board on 5 September 2026, so this row is unlikely to ever receive an independent score from that source. See /p/xiaomi-mimo-v2-6-pro-top-open-weights-model/. Re-checked 2026-10-16: vals.ai has not evaluated it; its SWE-bench Verified board is archived (88 systems, frozen since 2026-09-05). It lists MiMo V2.5 Pro (74.0%) and MiMo V2.5 (71.0%), which are earlier models and not an exact match for V2.6-Pro.",
      "sameAs": []
    },
    {
      "rank": null,
      "name": "Cohere North Mini Code",
      "maker": "Cohere",
      "open": true,
      "sweBenchVerified": null,
      "sweBenchVerifiedVendor": null,
      "sweBenchVerifiedIndependent": null,
      "independentSource": null,
      "independentDate": null,
      "rankedOn": null,
      "sweBenchPro": null,
      "terminalBench": null,
      "inputPricePer1M": "Free",
      "outputPricePer1M": "Free",
      "context": "256K",
      "verified": false,
      "announced": null,
      "source": {
        "title": "GENZ TECH — Cohere North Mini Code",
        "url": "https://genztech.blog/p/cohere-north-mini-code-open-h100-coder/"
      },
      "note": "Open-weight ~30B mixture-of-experts coder, free to use, runs on a single H100 with 256K context. Cohere published no SWE-bench Verified/Pro or independent coding eval, so it stays unranked pending confirmation (verifying). vals.ai has not run this model. Worth noting for expectations — the only Cohere model on its board, Command A, scores 7.8%, by far the lowest of the 80 systems tested, though that is a general-purpose model rather than this code-tuned one. Re-checked 2026-10-16: vals.ai's SWE-bench Verified board remains archived (frozen for posterity, no new models are being scored; 88 systems listed) and still shows no evaluation for this model; all 22 previously-tracked ranked scores were re-verified this run against the frozen table and are unchanged, exact match.",
      "sameAs": []
    },
    {
      "rank": null,
      "name": "KAT-Coder-Pro V2.5",
      "maker": "Kwaipilot (Kuaishou)",
      "open": false,
      "sweBenchVerified": null,
      "sweBenchVerifiedVendor": null,
      "sweBenchVerifiedIndependent": null,
      "independentSource": null,
      "independentDate": null,
      "rankedOn": null,
      "sweBenchPro": "65.2%",
      "terminalBench": null,
      "inputPricePer1M": "$0.74",
      "outputPricePer1M": "$2.96",
      "context": "256K",
      "verified": false,
      "announced": null,
      "source": {
        "title": "GENZ TECH — KAT-Coder-Pro V2.5 lands second on SWE-Bench Pro",
        "url": "https://genztech.blog/p/kat-coder-pro-v2-5-swe-bench-pro-second/"
      },
      "note": "Vendor-reported (KAT-Coder-V2.5 technical report, arXiv 2607.05471): SWE-bench Pro 65.2%, second only to Opus 4.8 at 69.2%, plus a best-in-test PinchBench 94.9% for agentic tool use, all run under a unified Claude Code harness. No SWE-bench Verified score published for V2.5, so it stays unranked pending independent confirmation. vals.ai has not evaluated it and does not cover Kwaipilot/Kuaishou at all, and llm-stats does not list the model. Pricing confirmed at $0.74/$2.96 per 1M (Pro) and $0.15/$0.60 (Air). Re-checked 2026-10-16: vals.ai's SWE-bench Verified board remains archived (frozen for posterity, no new models are being scored; 88 systems listed) and still shows no evaluation for this model; all 22 previously-tracked ranked scores were re-verified this run against the frozen table and are unchanged, exact match.",
      "sameAs": []
    }
  ]
}