A technical discussion highlights the 'acc vs acc_norm' problem in evaluating Large Language Models (LLMs), particularly in multiple-choice tasks. The standard accuracy metric (acc) favors shorter answers due to its reliance on summed log-likelihood, which is inherently penalized by answer length. The normalized accuracy metric (acc_norm) attempts to correct this by dividing the score by the byte length of the continuation, aiming for a more consistent comparison across models and tokenizers. However, acc_norm can also introduce biases, such as favoring answers that split into many predictable subwords. The article suggests reporting both metrics and choosing one consistently per task family before training to avoid misleading evaluations. AI
IMPACT Clarifies potential biases in LLM evaluation metrics, guiding researchers toward more accurate model comparisons.
RANK_REASON Technical paper discussing LLM evaluation metrics. [lever_c_demoted from research: ic=1 ai=1.0]
- HellaSwag
- Massive Multitask Language Understanding
- Piqan County
- reinforcement learning from human feedback
- supervised fine-tuning
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
