A new study published on arXiv introduces the "Prompt Lottery" phenomenon, revealing that even minor rephrasing of prompts used to evaluate large language models can lead to significantly different judgments. This sensitivity can even reverse the relative rankings of models. The research found that LLM-as-a-Judge evaluations are six to nine times less consistent under prompt rephrasing than human annotators, falling below usability standards. To address this, the study proposes using multi-prompt aggregation to improve evaluation robustness by focusing on consistent judgments across various prompt variants. AI
IMPACT Highlights a critical flaw in current LLM evaluation methods, potentially impacting model development and benchmarking.
RANK_REASON Academic paper detailing a new phenomenon in LLM evaluation. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
