A new arXiv paper titled "Similar Models, Broken Odds" investigates the limitations of the Bradley-Terry model used by Chatbot Arena to rank language models. The study found that the Bradley-Terry model, which assumes Luce's choice axiom, oversimplifies rankings by not adequately accounting for the decisiveness of votes between highly similar models. On over 1.8 million votes, votes between versions of the same model were 1.37 times more decisive than predicted, with similar effects observed for models within the same generation or from the same lab. The researchers propose a more accurate four-to-six parameter correction model that better predicts held-out votes, particularly by modeling ties, though it only slightly alters the overall leaderboard order. AI
IMPACT Highlights flaws in current LLM evaluation methods, potentially leading to more nuanced ranking systems.
RANK_REASON Academic paper analyzing LLM ranking methodology. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
