This paper introduces a statistical framework for evaluating large language models (LLMs) and multimodal large language models (MLLMs) using both direct human auditing and LLM-assisted methods. It proposes confidence-stratified sampling for direct auditing to improve accuracy estimation even with miscalibrated self-confidence. For LLM-assisted evaluation, the paper models the judge's performance using sensitivity and false-positive rates to derive an error-corrected estimator. Experiments on MMLU-Pro and MMMU-Pro demonstrate that while LLM-assisted evaluation can offer efficiency, its practical benefits are constrained by inference costs and the specific conditions under which it is applied. AI
IMPACT Provides a new statistical framework for evaluating LLMs and MLLMs, potentially improving the efficiency and accuracy of model assessments.
RANK_REASON Paper published on arXiv detailing a new statistical framework for LLM and MLLM evaluation. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
