Benchmarks

Artificial Analysis’s New A.I. Index Penalizes Open-Weight Models

co-produced by AI28. October 2026, 07:33

New tests, familiar winners and a rude awakening for open models: The independent benchmarking firm Artificial Analysis has released version 5 of its Intelligence Index, arguably the most closely watched ranking of A.I. models. Because the index now relies mostly on private test data, open-weight models, especially from China, have slid sharply. As a result, the gap between the best proprietary model and the best open model has widened from 3 points to 12.

What Changed in the Index

According to Artificial Analysis, it is the biggest update since the start of the year. The key change: About 60 percent of the index now relies on private test sets or held-out answers, which is meant to prevent models from simply being trained on known test questions. The index is made up of four equally weighted categories:

Knowledge work: Tasks like those in a real office, such as editing documents and handling automations.
Coding: New is FrontierCode 2.0, a private test from the A.I. company Cognition. A solution counts only if a maintainer would actually merge it.
Science: New is Terminal-Bench-Science, with 70 research tasks a model has to complete on its own, plus a particularly hard physics test.
General knowledge: A new, harder version of Humanity’s Last Exam and a test that penalizes hallucinations.

Among the tests dropped are the well-known coding test SciCode and the knowledge work test GDPval-AA.

The New Ranking

Little changes at the top, but a lot changes below it. Compared with the previous version’s scores, which Trending Topics reported recently, the picture looks like this:

Claude Opus 5.5 (Anthropic): Stays on top with 58 points.
GPT-6 Astra (OpenAI): Jumps from 53 to 57 points, taking second place.
GPT-6.1 Sol (OpenAI): Scores 55 points at roughly one-eighth of Opus’s cost.
Claude Sonnet 5.5 (Anthropic): Slips slightly from 56 to 54 points.
Gemini 4 Argon (Google): Drops from 53 to 50 points.
Muse Wonder Full (Meta): Falls sharply, from 56 to 50 points.

No model wins all nine individual tests. Opus 5.5 leads three of them.

Open Models Take a Hit

Open-weight models, whose weights are freely available so anyone can run them, are hit hardest:

Kimi K3.1 Pro (Moonshot AI): Falls from 55 to 46 points. It remains the best open model, although Moonshot has not yet released its weights.
DeepSeek V4.1 Flash: Plunges from 45 to 32 points.
MiMo-V2.6-Pro (Xiaomi): Loses 16 points, ending up at 30.
Muse Spark Open (Meta): The best open model from the U.S. sinks from 39 to 25 points.

The best model whose weights are already available now sits 26 points behind the leader. Even Anthropic’s small, inexpensive Claude Haiku 5.5, with 36 points, beats every open model with released weights. Europe’s great hope Mistral fares no better: Its not-yet-open preview model, Mistral Large 4 Preview, scores 25 points.

The biggest gaps show up in the new, harder tests. On the revised Humanity’s Last Exam, Kimi scores 37 percent and Opus 68 percent. On the scientific agent tasks, Kimi reaches 34 percent and GPT-6 Astra 63 percent, while most open models stay below 10 percent.

Testing at the Limit Gets More Expensive

The new tests demand much more from the models. Per task, models generate a median of about 59,000 output tokens, two-thirds more than before. That shows up in the costs: A run of Claude Opus 5.5 at its highest setting costs about $10 (about €9) per task, according to Artificial Analysis, compared with about $1.27 (about €1.13) for GPT-6.1 Sol. How much computing effort a model is allowed makes an enormous difference: At its lowest setting, Opus 5.5 scores only 35 points.

A Ranking Is Only as Good as Its Tests

For open models, the result is less clear-cut than it first appears. Kimi K3.1 Pro, for example, leads the entire field on the agent test Terminal-Bench 4.0, ahead of every proprietary model. The ranking also depends heavily on which tests a benchmarking firm chooses and how it weights them: In the previous version, Kimi K3.1 Pro was still in fourth place, just three points behind the leader. And because most of the test data is now secret, the results can no longer be independently verified from the outside. Artificial Analysis, by contrast, relies on secret test data precisely so that models cannot be trained on known questions.