Answer format significantly impacts gender bias measurement in LLMs, study finds

A new study published on arXiv investigates how variations in answer formats affect the measurement of gender bias in large language models. Researchers found that changing the format from closed-ended to Likert-scaled or open-ended responses can significantly alter bias measurements, sometimes even reversing outcome rankings. The study highlights that different formats elicit distinct model behaviors, such as forced-choice selections, scale-based distributions, or refusals in free-text generation. This underscores the importance of considering answer format as a critical factor in robust LLM evaluation. AI

IMPACT Highlights the need for standardized and multi-format evaluation to accurately assess LLM biases.

RANK_REASON Academic paper on LLM evaluation methodology. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• arXiv
• BBQ benchmark
• Franziska Weeber
• Gender Bias
• Large Language Models
• OpinionQA

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →