New Benchmark: Inclusion Arena Ranks LLMs by Real-World Use
The landscape of artificial intelligence is rapidly evolving, with new large language models (LLMs) emerging at a dizzying pace. For enterprises seeking to integrate these powerful tools, the challenge lies not just in identifying promising candidates but in understanding their true performance in real-world applications. While traditional benchmarks have been indispensable for initial evaluations, many rely on static datasets or controlled lab environments, often failing to capture how models truly interact with human users in dynamic, production settings.
Addressing this critical gap, researchers from Inclusion AI, an affiliate of Alibaba’s Ant Group, have introduced Inclusion Arena. This novel model leaderboard and benchmarking system shifts the focus from theoretical capabilities to practical utility, ranking LLMs based on actual user preferences in live applications. The core argument is straightforward: to genuinely assess an LLM, one must observe how people use it and how much they prefer its responses over others, moving beyond mere knowledge retention.
Inclusion Arena distinguishes itself from established leaderboards like MMLU and OpenLLM by integrating its evaluation mechanism directly into AI-powered applications. Unlike crowdsourced platforms, Inclusion Arena randomly triggers “model battles” during multi-turn human-AI dialogues within these real-world apps. Currently, the framework is integrated into two applications: Joyland, a character chat app, and T-Box, an educational communication app. As users interact with these applications, their prompts are invisibly routed to multiple LLMs, which generate responses behind the scenes. Users then simply choose the answer they like best, unaware of which model produced it. This direct, unbiased feedback forms the basis of the evaluation.
The system employs the Bradley-Terry modeling method for ranking, a probabilistic framework similar to the Elo rating system used in chess, which also underpins Chatbot Arena. While both Elo and Bradley-Terry are adept at inferring relative abilities from pairwise comparisons, the researchers assert that Bradley-Terry yields more stable ratings, providing a robust framework for assessing latent model capabilities. However, the prospect of exhaustively comparing a large and growing number of LLMs becomes computationally prohibitive. To overcome this, Inclusion Arena incorporates two innovative components: a placement match mechanism, which provides an initial ranking for newly registered models, and proximity sampling, which limits subsequent comparisons to models within a defined “trust region,” thereby maximizing information gain within a practical budget.
Inclusion AI’s initial experiments, drawing on data up to July 2025, comprised over 501,003 pairwise comparisons from more than 46,611 active users across the two integrated applications. The preliminary findings from Inclusion Arena indicate that Anthropic’s Claude 3.7 Sonnet, DeepSeek v3-0324, Claude 3.5 Sonnet, DeepSeek v3, and Qwen Max-0125 were among the top-performing models. While acknowledging the current dataset’s scope is limited to these two applications, the researchers aim to expand the ecosystem through an open alliance, anticipating that more data will lead to an even more robust and precise leaderboard.
The proliferation of LLMs makes it increasingly challenging for enterprises to select models for evaluation. Leaderboards like Inclusion Arena offer invaluable guidance to technical decision-makers, highlighting models that demonstrate superior performance in practical usage scenarios. While internal evaluations will always be crucial to ensure an LLM’s effectiveness for specific applications, these real-world benchmarks provide a clearer picture of the broader competitive landscape, helping organizations identify models truly aligned with their operational needs.