Researchers argue benchmarks miss much of collective model capability

A new arXiv paper argues that common LLM benchmarks understate real-world capability because they usually score one model on one run. The authors introduce a Capability Frontier that compares performance across model choices and sampled generations at different cost levels. Across 21 LLMs and 16 benchmarks, they report large gains from correcting for single-model and single-run assumptions, including an 82% improvement in one comparison. The claim is not that every deployment has an oracle selector; it is that benchmark headlines can miss the capability available to systems that route across models.

Key details: Submitted to arXiv on June 25, 2026, The paper studies 21 LLMs across 16 benchmarks, It models a Pareto frontier across model choices, generations, and cost, The authors report that single-run benchmark framing can understate collective model capability.

Why it matters: If routing and sampling change effective capability, benchmark leaderboards may be understating what deployed AI systems can actually achieve.