New study reveals widespread miscalibration in crowdsourcing aggregation methods

A new study published on arXiv investigates the calibration of crowdsourcing aggregation methods, finding that common techniques like Majority Vote and model-based aggregators can be severely miscalibrated. The research demonstrates that while the Bayesian aggregator is perfectly calibrated under its data-generating model, exact calibration for weighted majority votes is highly restrictive. Empirically, the study observed systematic miscalibration across numerous human and LLM-based crowdsourcing datasets. To address this, a novel post-hoc calibration procedure is proposed that aggregates multiple aggregators before applying temperature scaling, improving calibration and accuracy, particularly in high-redundancy scenarios. AI

IMPACT This research highlights potential inaccuracies in AI-driven decision-making processes that rely on aggregated human or LLM inputs, suggesting improvements for more reliable AI systems.

RANK_REASON Academic paper detailing novel research findings on a specific machine learning technique. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv stat.ML →

• alphaXiv
• arXiv
• Bayesian aggregator
• CatalyzeX Code Finder for Papers
• CORE Recommender
• CrowdFM
• DagsHub
• Gotit.pub
• Hugging Face
• Influence Flower
• ScienceCast

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →