Technology

OpenAI's GPT-6 Astra Sets New Records on ARC-AGI-3 Agentic Intelligence Benchmark

By Greg Kamradt

OpenAI's GPT-6 Astra model has achieved state-of-the-art results on ARC-AGI-3, a benchmark designed to test agentic intelligence in novel, turn-based environments, according to a new report.

Using a Standard harness, which allows a model to carry forward its own notes through an environment, Astra (max) scored 62.7% on the ARC-AGI-3 Semi-Private test at a cost of $26,000. With a Provider Adapter harness, which preserves opaque reasoning states between requests and uses compaction for longer conversations, Astra (high) achieved a 99.9% score for $19,000. Both results represent the best scores recorded on the benchmark's leaderboard.

ARC-AGI-3 evaluates four components of agentic intelligence: exploration, modeling, goal-setting, and planning and execution. Unlike earlier versions in the ARC-AGI series, it requires agents to explore unfamiliar environments, infer goals, and build internal models to plan actions without explicit instructions. Human participants can solve 100% of the benchmark's environments.

For cost comparison, human testers were paid $115 per 90-minute session plus $5 per completed game, averaging about $12.78 per attempted game. When accounting only for the brain's energy use as a proxy, calculated as electricity cost, this drops to roughly 0.067 cents per attempted game.

One notable finding was Astra's action efficiency. In tests using the Provider Adapter harness, Astra (max) required fewer actions than the human baseline on 96% of levels, using 51.7% fewer actions per level on average compared to a median human baseline established through testing of about 500 participants. Researchers noted this marks a milestone, as they had expected AI to require more exploratory actions than humans even when solving problems successfully.

Analysis of Astra's replay logs also revealed that the model developed a compact, code-like symbolic notation system to track game states, objects, coordinates, and multi-step plans. This custom shorthand allowed Astra to represent game mechanics as logical rules and record detailed information such as control mappings and player positioning across turns.

In a separate evaluation using the PRO-LONG harness, which grants Astra access to a sandboxed coding environment, researchers observed the model building custom tools for each game, including board parsers, planners, and persistent notes. In one maze-based game called tu93, Astra constructed several small software components, including a maze solver, a combat rules module, a patrol-tracking system, and a state-verification tool.

Researchers highlighted the difference between the two testing harnesses: the Standard harness offers a uniform, minimal interface across providers, while the Provider Adapter harness allows models to use context-management features specific to their design. Across 167 comparable test cases, Provider Adapter runs were about 3.66 times faster and used 49% fewer tokens than Standard harness runs.

Going forward, both Standard and Provider Adapter harness results will be reported on the ARC-AGI leaderboard, each clearly labeled. The organization's open-source testing repository documents both methods.

While Astra's performance marks what researchers describe as a significant step forward in frontier AI capability, they emphasized that solving ARC-AGI-3 does not constitute proof of artificial general intelligence. The benchmark's environments, while requiring causal reasoning and goal inference, remain deterministic and closed-ended, unlike the complexity of real-world scenarios. Researchers said they are now examining what future benchmarks should measure, including capabilities like recursive self-improvement and open-ended innovation.