A new paper highlights significant limitations in how Large Language Models (LLMs) estimate confidence for classification tasks. Researchers found that common methods, like verbalization, result in highly sparse confidence outputs, with models often producing only a few distinct confidence values. This sparsity critically impacts evaluation metrics, such as the Area Under the Accuracy-Rejection Curve (AUARC), where different interpolation methods can drastically alter model rankings. The paper proposes a new method, "verbalization logprobs," which weights verbalized digits by token probabilities to address sparsity and improve AUARC scores without increasing inference costs. AI
IMPACT Highlights critical evaluation challenges for LLM classification confidence, potentially impacting how model performance is assessed and compared.
RANK_REASON Paper detailing evaluation pitfalls and proposing a new method for LLM confidence estimates. [lever_c_demoted from research: ic=1 ai=1.0]
Read on Hugging Face Daily Papers →
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
