Researchers have developed FairCoder, a new benchmark designed to identify biases in large language models (LLMs) when they are used for high-stakes decisions like hiring or admissions. This benchmark frames decision-making tasks as coding problems, allowing for the systematic probing of implicit biases across various domains and fairness definitions. To address situations where LLMs might refuse requests, the study also introduces FairScore, a metric that evaluates both refusal behavior and the diversity of group outcomes. Initial experiments using a 1,000-sample dataset on advanced LLMs have revealed previously unrecognized bias patterns, such as favoring applicants from wealthier backgrounds in college admissions, highlighting the risks associated with deploying LLMs in critical decision-making roles. AI
IMPACT Highlights risks of LLM deployment in critical decision-making and provides a framework for future bias research.
RANK_REASON The cluster contains a research paper detailing a new benchmark and metric for evaluating LLM bias. [lever_c_demoted from research: ic=1 ai=1.0]
- alphaXiv
- arXiv
- CatalyzeX
- DagsHub
- FairCoder
- FairScore
- Gotit.pub
- Hugging Face
- LLM
- ScienceCast
- Yongkang Du
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
