GPT-6 Astra Scored 62.7% or 99.9%? The Benchmark Gap Explained

Benchmarks are supposed to settle arguments about which AI model is best. This week, one benchmark did the opposite: it showed that the same model can score 62.7 percent or 99.9 percent depending on how it is tested. The model is OpenAI's GPT-6 Astra, the test is ARC-AGI-3, and the gap between those two numbers has become a public argument about how AI capability should be measured at all.

ARC-AGI-3 is not a normal exam. It is a set of interactive games where an agent must discover the rules for itself, rather than answering questions it has seen before. It was designed to resist memorisation and brute-force tricks, which is why it has become a respected test of genuine reasoning. When ARC Prize first released it in March 2026, frontier models scored near zero. Humans cleared the same environments 100 percent of the time.

That history is what makes the new numbers so striking. On ARC Prize's own published leaderboard, GPT-6 Astra at its highest configuration leads with 62.7 percent. That is a genuine leap from near zero, and ARC Prize itself has described it as a major milestone. But OpenAI, when it launched Astra, published a figure of 99.9 percent. Both numbers come from the same model on the same benchmark family. The difference is the harness: the scaffolding wrapped around the model during the test.



Credit: OpenAI logo, via Converse360

Two harnesses, two different tests

Under ARC Prize's standard, provider-neutral harness, each request starts cold. The model keeps only the notes it chooses to preserve, and nothing else carries over between requests. That is the setup every lab's model must clear, which is why ARC Prize calls it the apples-to-apples comparison. On that harness, Astra at maximum reasoning effort scored 62.7 percent, at a reported run cost of $26,098.

Under OpenAI's provider adapter harness, Astra preserves an opaque reasoning state between requests and uses compaction for longer conversations, letting it reuse prior work instead of solving from scratch. On that harness, the same model scored 98.6 percent at maximum effort and 99.9 percent at high effort, at a reported cost of $18,817. Hold the model and the effort roughly fixed and change only the harness, and the score moves by 36 to 37 points.

ARC Prize's position is clear: the adapter result is not worthless, but it is not a comparable score. On the 167 pairs both harnesses solved, adapter runs were reported to be about 3.66 times faster and to use 49 percent fewer tokens. That is a real engineering advantage. But a model-plus-scaffolding result presented as a model result is a different claim, and it should not be used to rank one lab against another.

Why the harness matters so much here

The harness matters because ARC-AGI-3 is designed to test whether an agent can discover rules and remember what it has tried. A system that remembers its own reasoning state across requests has a structural advantage on exactly that kind of task. It is not cheating in the sense of seeing the answers, but it is a different kind of test: one that measures the model plus its memory system, rather than the model alone under neutral conditions.

That distinction also explains a wider pattern in recent benchmark news. In early October, an open-sourced harness called Prime Agent reportedly pushed public scores on ARC-AGI-3 from roughly 30 percent, where Claude Opus 5 had been sitting, up past 95 percent on the same weight class of model. Separately, a team at Nvidia reportedly got Claude Opus 5, the identical model that scores 30 percent by itself, to hit 100 percent just by rebuilding the software layer that pilots it through the game. The model did not get smarter. The code around it got better at remembering what it had already tried.

Better scaffolding is real progress, and it is how agents will actually be used. Nobody ships a bare model with no memory, tools or prompting. But a single headline number, without the harness named beside it, tells you very little about which model is actually better.

What OpenAI actually said

OpenAI's own footnotes acknowledged that its headline figure came from a harness adjusted from ARC Prize's original setup to better match real-world performance. That caveat was printed, but it did not travel with the number. Headlines and launch coverage carried 99.9 percent as if it were the model's score, full stop.

OpenAI's launch page also included a quote from ARC Prize Foundation's Greg Kamradt describing Astra as surpassing the organisation's human action-efficiency baseline on 96 percent of levels. That is a related but distinct metric from the headline percentage score, and it shows how easily one test run can generate several differently flattering statistics. Action efficiency measures whether the model used fewer actions than the human baseline, not whether it solved the same share of levels.

ARC Prize has also stated that saturating the benchmark would not represent proof of achieving AGI, citing its bounded scope and deterministic mechanics. That matters because the 99.9 percent figure was widely read as an AGI claim. The benchmark's own designers say it cannot support that claim, even at 100 percent.



Credit: Puzzle-game illustration, via Google Play

What this means for buyers and readers

For anyone choosing a model, the lesson is practical: never compare two numbers unless you know they were produced under the same harness, the same effort level and the same cost constraints. A model that scores higher under a generous harness may not be better for your task, and a model that scores lower under a neutral harness may be the stronger choice once you add your own scaffolding.

For investors, the gap is a warning about pricing capability. If a headline score can swing by nearly 40 points without changing a single weight, then a company's valuation cannot rest on that number alone. The question to ask is not "what did it score?" but "what did it score, under what setup, at what cost, and can I reproduce it?"

For the AI industry, the argument is healthy, even if it is uncomfortable. Benchmarks only work if everyone trusts the rules. When a lab publishes a number under its own harness and presents it as comparable to neutral results, trust erodes, and the next benchmark becomes harder to believe. ARC Prize's decision to publish both numbers, with the harness named, is the right model: transparency about setup, not just about scores.

The wider benchmark problem

ARC-AGI-3 is not the only benchmark with this problem, but it is the clearest example. Many benchmarks now depend heavily on prompting, tools, memory and multi-step scaffolding. Those are legitimate parts of a real system, but they are not part of the model. When a launch page reports a system score as a model score, it hides where the improvement actually came from.

The fix is not to ban scaffolding. It is to label it. A score should travel with its harness, effort level, cost and date, like a scientific result travels with its method.

Until that becomes standard, the safest way to read any AI benchmark claim is to ask three questions: who ran it, under what setup, and what did the neutral run say? In Astra's case, the neutral run says 62.7 percent, which is still a remarkable result. It just is not 99.9 percent, and it is not AGI.



Credit: Puzzle-game illustration, via Google Play

Key Takeaways

  • On ARC Prize's standard, provider-neutral harness, GPT-6 Astra scored 62.7 percent on ARC-AGI-3 at maximum reasoning effort, at a reported $26,098 per run.
  • On OpenAI's provider adapter harness, the same model scored 98.6 percent at maximum effort and 99.9 percent at high effort, because it preserves reasoning state between requests.
  • ARC Prize says the adapter result is a model-plus-scaffolding result, not a comparable model score, and should not be used to rank labs against each other.
  • ARC-AGI-3 was designed to resist memorisation, and frontier models scored near zero when it launched in March 2026, so 62.7 percent is still a major milestone.
  • ARC Prize has stated that saturating the benchmark would not prove AGI, given its bounded scope and deterministic mechanics.
  • Readers should compare benchmark numbers only when the harness, effort level, cost and date are the same — otherwise they are comparing systems, not models.

The Astra benchmark gap is not really about one model. It is about what a score means. A model that scores 62.7 percent under neutral conditions and 99.9 percent with its own memory system is not two different models. It is one model, measured two ways. The industry needs to start saying which way, every time, or the numbers will stop meaning anything at all.

Sources

Comments

Post a Comment