Research Questions Fairness of LM Arena’s AI Model Evaluation Practices

A recent study led by researchers from Cohere, Stanford, MIT, and the Allen Institute for AI (Ai2) has raised serious concerns about LM Arena, the organization behind Chatbot Arena — a leading AI model benchmarking platform. The study alleges that LM Arena enabled major AI companies such as Meta, OpenAI, Google, and Amazon to privately test multiple model variants and selectively disclose only the best-performing ones. This practice reportedly gave these companies an advantage on the public leaderboard, while others were not informed about or offered the same opportunity. Notably, Meta is said to have tested 27 model variants privately before the release of Llama 4 but revealed only the top-performing one. The study, which analyzed over 2.8 million model comparison battles, also found that some companies benefited from higher sampling rates, increasing their exposure and possibly improving performance on related benchmarks. LM Arena has disputed the study’s findings, calling them inaccurate and questioning the methodology, particularly the reliance on AI model self-identification. The researchers argue for more transparency and suggest policy changes such as public disclosure of all test results and equal sampling for all participants. The controversy intensifies scrutiny on the impartiality of AI benchmarks amid rising commercial stakes and competition in the sector.