Meta Releases Tuned AI Model for Benchmark Testing, Raising Transparency Concerns

Meta has recently launched a new flagship AI model called Maverick, which ranked second on LM Arena — a platform where human raters evaluate AI model outputs. However, questions have arisen about the transparency of this ranking. According to Meta’s own disclosures, the version of Maverick used in LM Arena testing is a fine-tuned experimental chat version optimized specifically for conversational performance. This differs from the general version released to developers. This discrepancy has sparked criticism among AI researchers who argue that such practice misleads users and developers about the model’s real-world capabilities.

Critics point out that tailoring models for benchmarks while distributing less capable variants publicly undermines trust and makes it harder to gauge actual performance. The LM Arena version of Maverick reportedly uses excessive emojis and long-winded responses — characteristics not observed in the publicly downloadable version. Researchers argue that benchmarks should represent a model’s typical behavior rather than an enhanced or selectively modified version. While fine-tuning for benchmarks is not new, openly releasing differing versions raises ethical concerns around transparency and user expectations.

Meta and the organization behind LM Arena, Chatbot Arena, have been contacted for comment. The incident calls into question the reliability of popular benchmarking platforms like LM Arena, especially when companies alter models to perform better in such controlled evaluations.