Medical AI models inject unrequested patient demographics with DEI prompts

A new study published on arXiv has identified a phenomenon called "demographic injection" in medical language models. When prompted with diversity, equity, and inclusion (DEI) instructions, these models tend to add unrequested demographic attributes like race, socioeconomic status, or sex to patient scenarios, even when such details are not present in the original query. This effect, observed across 47 different models, significantly increases the rate of demographic injection from 0.7% to 33.1%. While most injected details are general statements, a small percentage alter the model's recommended course of action, leading to incorrect outputs. AI

IMPACT This research highlights a critical flaw in current medical AI models, suggesting that DEI prompts can inadvertently lead to inaccurate patient representations and potentially flawed clinical recommendations.

RANK_REASON The cluster contains a research paper detailing a new phenomenon observed in AI models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• Demographic Injection
• Diversity, Equity, and Inclusion

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →