LLM confidence estimates flawed by sparsity, new paper finds

A new research paper published on arXiv highlights significant limitations in how large language models (LLMs) estimate confidence for classification tasks. The study found that common methods like verbalization produce overly sparse outputs, with models like Qwen3-32B offering very few unique confidence values. This sparsity critically impacts evaluation metrics, altering model rankings based on interpolation choices. The researchers propose a new method called 'verbalization logprobs' which addresses sparsity and improves confidence estimation without increasing inference costs. AI

IMPACT Highlights critical issues in LLM confidence estimation, potentially impacting reliability in classification tasks and influencing future evaluation methodologies.

RANK_REASON Research paper published on arXiv detailing evaluation methods for LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• alphaXiv
• arXiv
• AUARC
• CatalyzeX
• DagsHub
• Gotit.pub
• Hugging Face
• Qwen3 32B
• ScienceCast
• SST-2 Benchmark
• verbalization logprobs

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →