The Use of ‘Vibes’ as a Subjective Measure in Evaluating AI Progress

The article explores a recent trend in the artificial intelligence (AI) community where subjective perceptions, or ‘vibes’, are being used as informal indicators of progress in generative AI and large language models (LLMs). Traditionally, AI systems have been assessed using quantitative metrics like Time to First Token (TTFT) or Tokens Per Second (TPS). However, some AI leaders, including OpenAI’s CEO Sam Altman, have begun referencing how certain models ‘feel’ during interactions. Altman’s posts on X (formerly Twitter) mention how GPT-4.5 provides a ‘thoughtful’ interaction and possesses a ‘magic’ that defies traditional benchmarks. This approach, while appealing to those who argue that AI should be evaluated more holistically—similar to how we judge humans—has sparked strong criticism. Detractors argue that relying on vibes undermines scientific rigor and fuels anthropomorphism of AI, potentially muddying the waters for progress assessment. The article warns that if the trend gains wider acceptance, it could erode objective standards in AI research. Despite the controversy, some researchers believe that vibes offer a visceral understanding of AI’s capabilities, particularly in areas like emotional resonance or creativity, which are difficult to quantify but increasingly relevant in generative systems.