Kimi K3 Needed 10x the Tokens to Tie Fable 5 on SWE

Fireworks AI published a head-to-head evaluation hours ago that ran Kimi K3 and Claude Fable 5 through roughly 1,030 tasks inside real agent loops, and on software engineering the two finished level: 92.4% for K3, 92.6% for Fable 5. The number that deserves more attention sits underneath that tie. On the same SWE tasks, K3 needed about 55 turns and 1.3 million tokens. Fable 5 needed about 21 turns and 130,000 tokens. Identical outcome, roughly ten times the tokens.

• The eval is big and agentic, not single-shot. Five task families, about 1,030 tasks total: 460 software engineering, 225 multi-language across six languages, 120 lawyer-graded legal tasks, 100 algorithmic problems and 89 terminal tasks.
• The scores are a tie, the effort is not. K3 grinds through SWE tasks with far more turns and tokens; on long terminal work the pattern inverts and Fable 5 is the one burning 64 turns and 1.5 million tokens, sometimes running into timeouts.
• The 93% "state of the art" figure is an oracle result. Fireworks defines it as running each task through both models and keeping the cheapest correct answer. That is a ceiling on what routing could buy, not a router you can deploy.
• The publisher is not neutral. Fireworks serves K3 on its own platform and quotes the cost advantage at its own prices. It is a useful dataset, not an independent referee.

What did Fireworks actually measure?

Not a leaderboard submission. Fireworks assembled five families and ran each task inside an agent loop, so the model had to plan, call tools, read results and retry rather than emit one patch. The families: 460 software engineering tasks, 225 multi-language tasks spanning six languages, 120 legal tasks graded by lawyers, 100 algorithmic problems in the LeetCode and AtCoder mould, and 89 terminal tasks covering long agentic operations.

RelatedClaude Sonnet 5: Near-Opus Coding at Half the Price

Outside SWE, the results are close enough that Fireworks describes them in words rather than numbers. Fable 5 takes Java, Python and C++ on the multi-language set; K3 pulls level on JavaScript and Rust. Legal and algorithmic land within a few points. The one family with a clear winner is terminal, where K3 solved 11 tasks that Fable missed against 7 the other way, with a cluster of security and cryptography problems driving the gap.

One gap in the writeup matters: Fireworks does not name the harness or scaffold it used. SWE-style scores measure the model and the code driving it, which is exactly why the same model can post 88.6% on one harness and 85.8% on another. Without the scaffold, nobody can reproduce these numbers.

Why does Kimi K3 need ten times the tokens?

Turn count is the whole answer. At 55 turns against 21, K3 is re-reading its context, re-running tests and retrying roughly two and a half times as often, and every extra turn re-sends the accumulated transcript. Token spend compounds with turns, so a 2.6x gap in steps becomes a 10x gap in tokens. K3 solves the problem by grinding. Fable 5 solves it in fewer, larger, better-aimed steps.

The interesting part is that this is not a fixed property of either model. On terminal tasks the profile inverts: Fable 5 is the one taking 64 turns and 1.5 million tokens, and Fireworks notes it sometimes runs long enough to hit timeouts. A model that plans in big steps does well when the task is legible from the repository and badly when it has to feel its way through an unfamiliar shell. The two models do not differ in raw capability so much as in where they get stuck.

Does the cost claim survive the token count?

Fireworks says K3 can be "up to ~50X more cost effective than Fable alone on long agentic loops," and separately that K3 came out cheaper across all five families. The first claim is well scoped: on terminal work, where Fable burns 64 turns to K3's fewer, a large cost gap is exactly what you would expect.

The second claim is harder to square with the numbers in the same post. At the list prices we reported on July 19, K3 runs $3 per million input tokens against $10 for Fable 5, a bit more than a 3x per-token advantage. Ten times the tokens against a 3x per-token discount does not obviously net out in K3's favour on the SWE family. Fireworks did not publish the per-family cost table that would settle it, and its cost comparison is quoted on its own platform at its own prices rather than at either vendor's list rate. Until that table exists, treat "cheaper everywhere" as unverified and "much cheaper on long loops" as the supported version.

What is an oracle router, and why is 93% not a product?

The line getting quoted from this post is that K3 plus Fable 5 together reach 93% and beat either model alone. Fireworks is refreshingly direct about how that number was produced: oracle routing, which it defines as running the task through each model, then picking the cheapest correct option. Reaching for the cheapest correct answer requires already knowing which answers were correct, which requires running and grading both models on every task.

That is a measurement of the ceiling, not a system. Fireworks says so in the post: in a practical router you do not get to run your task against multiple models, and building one that comes close will need an order of magnitude more routing data. Plenty of vendor benchmark posts would have quietly dropped that caveat. This one leads with it, and the honest reading of the 93% is "here is what perfect model selection would have been worth on this task mix," which is roughly half a point over the better single model.

The distribution is the more useful finding. The oracle sent 72% to 96% of tasks to K3 depending on family, reserving Fable 5 for a specialised minority. If a real router could recover even part of that, the economics of running a frontier-adjacent open-weight model as the default and escalating to a closed model on hard cases start to look serious.

How does this square with our leaderboard?

Our AI coding leaderboard carries vals.ai's independent run, which puts K3 at 93.4% ±1.11 on SWE-bench Verified and Fable 5 at 95.0%, a gap that sits inside the combined error bars. Fireworks used a different harness and a different task mix, so its 92.4 against 92.6 is not directly comparable. Both readings point the same direction: on software engineering these two models are difficult to separate.

RelatedKimi K3 Is Third Best at Coding. It Is Also Not Open.

What we are not doing is moving the board on this post. Fireworks serves K3 on its own platform and quoted the cost advantage there; it raised a $1.5B Series D at a $17.5B valuation on July 17. That does not make the data wrong, and running 1,030 agentic tasks is real work that no neutral evaluator has published. It does mean the numbers are vendor-adjacent, and vendor-adjacent figures have run 2.6 to 11.6 points above neutral harnesses in our own tracking. Our board only ranks scores confirmed against an independent evaluation.

Worth restating for anyone reading "open model beats closed model" into this: K3's weights are still not downloadable. Moonshot has promised them by July 27, 2026, with no licence stated. Everyone benchmarking K3 today, Fireworks included, is hitting an API.

What it means for the market

The signal here is that model choice is turning into a per-workload decision rather than a per-company one. If K3 wins terminal work and Fable 5 wins Java refactors, no single API contract is optimal, and the value shifts toward the layer that can host both and decide between them. That favours multi-model inference platforms, Fireworks among them, and it puts slow pressure on the assumption that a frontier lab captures a customer's entire agent workload.

For Anthropic the read is more specific than a price threat. Fable 5's advantage in this data is turn efficiency, not accuracy, and turn efficiency is worth real money when agents run unattended for hours. The thing to watch is whether Anthropic starts marketing tokens-per-solved-task rather than benchmark percentages, because on that metric it currently looks strongest exactly where the headline scores say it is tied. None of this is investment advice; it is where the operating leverage sits.

Our take

The tie is not the story, and "open model catches closed model" is not the story either, partly because K3 still is not open. The story is that two models can post the same score on the same 460 tasks while spending an order of magnitude apart to get there, and that which one is the wasteful one flips depending on the task family. Benchmarks that report only a solve rate hide that completely.

Fireworks deserves credit for publishing turn counts and token counts at all, and for labelling its own headline figure as an oracle result instead of letting it travel as a product claim. The gap in the work is the missing per-family cost table and the unnamed harness, and those are exactly the two things that would let someone outside Fireworks check whether "cheaper in all five categories" holds. Until they land, the durable finding from this evaluation is not 93%. It is 55 turns against 21.

Original analysis by GenZTech. Figures from Fireworks AI's published evaluation, with independent scores from vals.ai.