New algorithm learns worker reliability and item rewards from comparisons

Researchers have developed a new algorithm to learn item rewards and worker reliability simultaneously from pairwise comparisons, particularly in crowdsourcing contexts. The proposed method utilizes a Boltzmann-rational model, extending the Bradley-Terry-Luce model by incorporating worker competencies. An EM-based algorithm is derived using Polya-Gamma latent variables, transforming the problem into a matrix sensing task with theoretical convergence guarantees. Experiments on synthetic and real-world data show the algorithm's robustness against unreliable and adversarial workers, outperforming existing baselines. AI

IMPACT This research could improve the quality of data used for training AI models by better accounting for unreliable human annotators.

RANK_REASON The cluster contains an academic paper detailing a new algorithm for learning from pairwise comparisons. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• Amazon Mechanical Turk
• Boltzmann-rational model
• Bradley-Terry-Luce model
• EM-based algorithm
• Kaustubh Shivshankar Shejole
• Polya-Gamma
• Scale AI

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →