A new metric called the Slop Composite Index has been developed to evaluate the "AI-ness" of large language models. This index assesses 39 frontier models by having them complete various tasks, such as writing cold emails and tweets. The outputs are then rated by a combination of human evaluators and by comparing them against human-written text from before 2022, including Enron emails and old tweets. Surprisingly, the results indicate that larger, more advanced models tend to exhibit more "slop" or artificiality than smaller, less expensive models, with common deductions for templated openings, repetitive structures, and overuse of dashes. AI
IMPACT This new metric could influence how AI models are developed and evaluated, potentially leading to more natural-sounding AI outputs.
RANK_REASON The cluster describes a new evaluation metric and its results for LLMs, which constitutes research. [lever_c_demoted from research: ic=1 ai=1.0]
Read on Mastodon — mastodon.social →
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
