Researchers have developed a new metric, the pooled selection-rate standard deviation (pooled SD), to analyze option bias in multiple-choice evaluations for AI models. This metric quantifies how unevenly option letters are selected across different question orderings. The pooled SD is composed of two factors: the flip rate, which measures how often a model's chosen option changes with reordering, and the positional skew, which indicates the standard deviation of option-letter selection rates. Experiments across 41 checkpoints and 12 benchmarks showed that the flip rate correlates with accuracy and output uncertainty, while positional skew relates to the estimated option-symbol prior. Mitigation strategies like prior correction and aggregation were found to affect these factors differently, highlighting the need to report both to fully understand model behavior. AI
IMPACT Introduces a new metric to better understand and mitigate biases in AI model evaluations, potentially leading to more reliable assessments.
RANK_REASON The cluster contains a research paper detailing a new metric for evaluating AI model bias. [lever_c_demoted from research: ic=1 ai=1.0]
- alphaXiv
- arXiv
- CatalyzeX Code Finder for Papers
- Connected Papers
- DagsHub
- Gotit.pub
- Hugging Face
- Influence Flower
- Litmaps
- ScienceCast
- scite Smart Citations
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
