Are we chasing ghosts? Quantifying unattributable polarization, and attributing the rest to annotator groups
Researchers have developed a new metric and an open-source Python library to better quantify and attribute polarization in subjective NLP datasets. Existing methods struggle with inherent polarization and canceling effects, but the new approach identifies statistical significance of polarization attributed to specific annotator groups. Applying this to four datasets revealed that gender and race consistently explain polarization patterns, with differences intensifying as groups diverge. AI
IMPACT Provides a more robust method for evaluating subjective NLP tasks, potentially improving the reliability of models trained on such data.