Small data changes can flip top LLM rankings, study finds

A new research paper proposes a method to evaluate the robustness of large language model (LLM) ranking systems. The study found that removing a very small percentage of preference data, as little as 0.003%, can significantly alter the rankings of top-performing models on platforms like Chatbot Arena. The research also noted that rankings derived from MT-bench preferences are more stable than those from Chatbot Arena, likely due to MT-bench's use of expert annotators. The paper concluded that both crowdsourced human evaluations and LLM-as-a-judge preferences exhibit similar sensitivity to data removal. AI

IMPACT Highlights potential instability in LLM leaderboards, suggesting a need for more robust evaluation methods.

RANK_REASON Academic paper detailing a new evaluation method for LLM ranking systems. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv stat.ML →

• Bradley--Terry model
• Chatbot Arena
• Large Language Model
• LLM-as-a-judge
• MT-bench

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →