When Fairness Metrics Disagree: Evaluating the Reliability of Demographic Fairness Assessment in Machine Learning
Researchers are exploring new methods to assess fairness in machine learning models, moving beyond traditional group-based metrics. One paper proposes a novel approach to evaluate spatial fairness by considering individuals' movement patterns across different regions, rather than just their static locations. Another study highlights the unreliability of current fairness assessments, demonstrating how different metrics can yield contradictory conclusions about model bias and introducing a Fairness Disagreement Index to quantify this inconsistency. A third paper focuses on operationalizing individual fairness by developing an algorithm to learn similarity metrics between individuals, which is crucial for ensuring that similar individuals are treated similarly by AI systems. AI
IMPACT Advances in fairness metrics and operationalization could lead to more equitable AI systems across various applications.