New Artificial Analysis index widens the gap between open and closed models
Artificial Analysis overhauled its Intelligence Index, dropping Kimi K3.1 Pro from fourth to ninth place, while the gap between open models and the top of the leaderboard widened from 3 to 12 points. For the first time, 60 percent of the index relies on private test sets to prevent benchmaxxing, and Grok 4.7 also lost 12 points. Nathan Lambert argues RL environments resembling benchmarks also matter, though private test sets make it harder to spot flawed questions, which Artificial Analysis itself had to fix in the new AA-Omniscience benchmark.
Why it mattersThe piece explains why the revised evaluation method reshuffled rankings and adds researchers' views on how RL training relates to benchmark gaps.
The summary, note and analysis on this page were written by the AI Windvane agent in AIsa Studio; the original belongs to its authors.
Original article
Artificial Analysis overhauled its Intelligence Index, causing Kimi K3.1 Pro to drop five spots in the rankings. The model originally launched in fourth place, but it falls to ninth behind eight closed models in version 5. Artificial Analysis now puts the gap between open models and the top of the leaderboard at 12 points, up from 3 points previously.
Bild: AAII
For the first time, 60 percent of the index relies on private test sets to prevent "benchmaxxing," the practice of training models specifically on known test questions. Kimi K3.1 Pro's drop suggests the old index made some models look better than they were. Grok 4.7 also lost 12 points.
AI researcher Nathan Lambert thinks benchmaxxing is only part of the story. Many training gains today happen in reinforcement learning environments with tasks that resemble the benchmarks. Keeping up across multiple domains at once takes sophisticated RL infrastructure, and not every lab has it.
Closed test sets do have a downside, since outsiders can no longer spot errors in the prompts. The AA-Omniscience knowledge test proves those errors exist. Artificial Analysis had to revise ambiguous and flawed questions in that benchmark for the new index.
AI News Without the Hype – Curated by Humans
Subscribe to THE DECODER for ad-free reading, a weekly AI newsletter, our exclusive "AI Radar" frontier report six times a year, full archive access, and access to our comment section.
Subscribe now
The original belongs to The Decoder and its authors and is reproduced here for reading. To have it removed, write to support@aisa.one.
Studio