New ConfBench benchmark evaluates VLM confidence in document extraction

A new benchmark called ConfBench has been developed to assess the trustworthiness of confidence scores in vision-language models (VLMs) for document extraction tasks. The benchmark, which includes degraded document samples, reveals that while model capability is a key factor in confidence accuracy, the specific VLM family and modality (OCR+Image) also play significant roles. Post-hoc correction methods can improve calibration for routing extractions, and log-probability with first-token aggregation appears to be the most effective confidence estimation method. AI

IMPACT This benchmark will help developers build more reliable AI systems for document processing by improving confidence estimation in vision-language models.

RANK_REASON The cluster contains an academic paper introducing a new benchmark for evaluating AI models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• arXiv
• Claude
• ConfBench
• DagsHub
• Hugging Face
• optical character recognition
• vision-language model

• paper
• product

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →