Gemini 4 Carbon Takes the MathArena Lead From GPT-6.1 Sol
Google's Gemini 4 Carbon has taken first place on MathArena, the third-party benchmark that tests AI models on fresh mathematics, scoring 85.4% overall expected performance ahead of OpenAI's GPT-6.1 Sol at 80.6% and Anthropic's Claude Fable 5.2 at 82.8%. Jasper Dekoninck, the ETH Zurich researcher who runs MathArena, announced the change on X on 13 November, writing that "for the first time in a very long time, we have a non-OpenAI model as #1 on MathArena." It is a real milestone for Google — and a lead that comes with two large asterisks: the gap to second place is inside the margin of error, and Carbon costs about fifteen times as much per problem as the model it displaced.
MathArena is a benchmark built to stop models from winning by memorisation. It evaluates large language models on newly released mathematics competitions, olympiads and research problems, so the questions are ones the models have not seen during training. Each model attempts every problem four times, and the leaderboard shows the average score and the average cost of a single run on one problem. The project comes from ETH Zurich and INSAIT, was published at NeurIPS 2025, and its code and data are open source.
What the leaderboard actually says
The overall table on matharena.ai, as read on 14 November, ranks the models by expected performance — the average of normalised scores across all questions from the competitions currently in use, with each question given equal weight. Carbon, run at its "high" reasoning setting, leads at 85.4%. Fable 5.2, at its deepest "max" setting, is second at 82.8%. GPT-6.1 Sol, also at "max", is third at 80.6%. The best openly licensed model is Moonshot AI's Kimi K3.1 Pro at 54.3%, a long way behind the three closed frontier models.
| Model | Reasoning setting | Expected performance | 95% confidence interval | Cost per problem |
|---|---|---|---|---|
| Gemini 4 Carbon (Google) | High | 85.4% | ±3.0 points | $15.70 |
| Claude Fable 5.2 (Anthropic) | Max | 82.8% | ±5.3 points | $40.20 |
| GPT-6.1 Sol (OpenAI) | Max | 80.6% | ±2.7 points | $1.07 |
| Kimi K3.1 Pro (Moonshot AI) | Max | 54.3% | ±5.8 points | $3.70 |
The table's own numbers argue for caution about the headline. MathArena reports a 95% confidence interval of plus or minus 3.0 points for Carbon and plus or minus 5.3 points for Fable. The 2.6-point gap between first and second place falls inside that range, which means the data does not firmly settle the order between Carbon and Fable. The gap to Sol, at 4.8 points, is clearer. Carbon's lead is real, but it is not decisive over its nearest rival.
![]() |
| AI-generated illustration (generated 14 Nov 2026) |
The timing also matters. Results for Carbon and Fable 5.2 were added to MathArena on 13 November, days after both models shipped: Carbon reached general availability on 10 November, and Anthropic released Fable 5.2 on 9 November. Sol, released on 29 September, had held the top spot until then. Dekoninck's announcement was the news; Google itself has not announced the ranking, and both Google chief executive Sundar Pichai and Google DeepMind chief executive Demis Hassabis reposted Dekoninck's post without adding a comment of their own.
The cost of the lead
The second asterisk is the price. On MathArena's own cost figures, solving one problem once costs $15.70 with Carbon, about fifteen times Sol's $1.07. Fable is more expensive still at $40.20 per problem — roughly thirty-eight times Sol. Kimi K3.1 Pro, the best open model, costs $3.70. In other words, Carbon bought its 4.8-point lead over Sol with roughly fifteen times the spending per problem, and Fable spent far more than either for second place.
Dekoninck said so himself in a follow-up post, writing that "the extra performance does come at significantly higher costs under these reasoning settings," and pointing to BrokenArXiv, one of the benchmark's evaluations, where the gap is particularly visible. The cost problem also limited what was measured at all. Because of cost, Carbon was run only on ArXivLean, where Dekoninck said its performance was "very impressive", and it will be added to LeanLean later. For evaluations where a model has no results, MathArena estimates the score statistically from the results of other models and problems and includes the estimate in the average. Carbon's headline number therefore mixes measured results with modelled estimates — something the benchmark discloses, but a headline rarely mentions.
Why this benchmark is worth taking seriously
Most AI benchmarks have a contamination problem: if a test's questions were in a model's training data, a high score may show memory rather than reasoning. MathArena's answer is to use competitions as they are released, so the problems are new by construction. Its NeurIPS paper reports strong signs of contamination in AIME 2024, a widely used maths test.
The benchmark also tests something most others do not: writing proofs. Many maths tests only check a final answer. MathArena includes proof-writing, where a model must produce a chain of reasoning a human can check. On IMO 2025, the best models scored slightly less than 40%, which shows how much headroom remains even at the frontier. A leaderboard lead on MathArena is therefore a narrower, more specific claim than "best at maths" in general — it is best at fresh competition problems and research-level questions, under these settings, at these costs.
![]() |
| AI-generated illustration (generated 14 Nov 2026) |
Background: Gemini 4 Carbon is Google's strongest frontier model, announced on 28 October and generally available from 10 November. Our flagship AI model comparison puts Carbon against Claude Opus 5.5 and GPT-6 Astra, while the GPT-6 Astra benchmark scores explain why one model's number can look wildly different on two tests. The best open model on MathArena, Kimi K3.1 Pro, trails the closed leaders by more than 25 points.
What this changes — and what it does not
For Google, the result is a genuine talking point after a year in which OpenAI models have usually topped the maths boards, arriving in the same week Carbon went generally available. For Anthropic, second place at the deepest reasoning setting, and the highest cost on the board, raises a different question: whether Fable's "max" setting is worth $40.20 a problem outside research use. For OpenAI, third place is not a defeat: Sol remains by far the cheapest of the three per problem, and it is built for everyday agentic coding and professional work rather than maximum-effort reasoning. A leaderboard measures one thing; a product decision weighs score, cost, speed and fit together.
What the result does not do is settle which model is "best at maths". The settings differ — Carbon at high, its rivals at max — the Carbon–Fable gap is inside the confidence intervals, part of Carbon's score is statistically estimated rather than measured, and the costs differ by an order of magnitude. The honest reading is narrower and still interesting: on fresh, hard maths problems, Google's most expensive reasoning setting now edges out the field, at a price most users would not pay per problem.
![]() |
| AI-generated illustration (generated 14 Nov 2026) |
Key Takeaways
- Gemini 4 Carbon leads MathArena at 85.4% expected performance, ahead of Claude Fable 5.2 at 82.8% and GPT-6.1 Sol at 80.6% — the first non-OpenAI model to top the board in a long time.
- The 2.6-point lead over Fable sits inside the 95% confidence intervals (±3.0 for Carbon, ±5.3 for Fable), so the order of the top two is not statistically decisive.
- Carbon costs $15.70 per problem, about 15 times Sol's $1.07; Fable costs $40.20, about 38 times Sol's — the leaderboard lead was bought with far higher spending.
- Carbon was run only on ArXivLean because of cost, and scores for unrun evaluations are statistically estimated; Google has not announced the ranking, and Pichai and Hassabis reposted the operator's post without comment.
- MathArena tests fresh competition and research problems to avoid contamination, and is the first benchmark to test proof-writing — on IMO 2025 the best models scored under 40%.
Conclusion
Gemini 4 Carbon's MathArena lead is real, verified on the benchmark's own table, and genuinely notable as the first non-OpenAI model to top the board in a long time. It is also a qualified lead: narrow enough to sit inside the margin of error against Fable, expensive enough to cost fifteen times its predecessor per problem, and partly estimated rather than fully measured. Carbon is now the model to beat on hard, fresh mathematics, if you can afford the reasoning bill. For everyone else, Sol's third-place score at a fifteenth of the cost remains the more practical number. Watch the next MathArena update, when Carbon is added to LeanLean, for a fully measured score.
Sources
- MathArena — overall leaderboard (Carbon 85.4%, Fable 5.2 82.8%, GPT-6.1 Sol 80.6%, Kimi K3.1 Pro 54.3%; costs per problem)
- ai-oriens — Operator of third-party math benchmark MathArena says Gemini 4 Carbon is now No. 1 (13–14 Nov 2026; quotes Dekoninck's X post and the site table)
- MathArena: Evaluating LLMs on Uncontaminated Math Competitions (NeurIPS Datasets and Benchmarks 2025; Balunović, Dekoninck, Petrov, Jovanović, Vechev)



Comments
Post a Comment