New framework combines human judgment and AI scores for better assessments

Researchers have introduced Aggregate-then-Calibrate (AtC), a novel two-stage framework designed to improve human-centered assessment tasks. This method combines heterogeneous human judgments, accounting for annotator reliability, with model-generated scores. AtC theoretically demonstrates that modeling annotator heterogeneity leads to more efficient consensus estimation and that its isotonic calibration offers risk bounds even with misspecified consensus rankings. Empirical results show AtC consistently enhances accuracy and robustness compared to assessments relying solely on human or model inputs. AI

IMPACT This framework could improve the reliability and accuracy of AI-assisted decision-making processes in fields requiring human judgment.

RANK_REASON The item is an academic paper detailing a new framework for assessment tasks. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv stat.ML →

• Aggregate-then-Calibrate
• alphaXiv
• Anatomical Therapeutic Chemical Classification System
• arXiv
• CatalyzeX
• DagsHub
• Gotit.pub
• Hugging Face
• Human-centered assessment
• IArxiv
• isotonic projection
• rank-aggregation model
• ScienceCast

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →