A new benchmark called INCLUDE has been developed to assess socio-cultural biases in Large Language Models (LLMs) across various Indian languages. Current safety alignment for LLMs is primarily English-focused, leading to potential harm when these models are used in multilingual contexts. The INCLUDE benchmark, comprising 2,604 prompts in English, Hindi, Bengali, Marathi, Tamil, and Hinglish, was used to evaluate ten LLMs. Results indicated that Bengali showed the highest average bias in open-source models, while English exhibited the lowest bias in open-source models but the highest in closed-source models. AI
IMPACT Highlights critical safety gaps in LLMs for non-English languages, potentially impacting global AI deployment and user trust.
RANK_REASON The cluster contains an academic paper introducing a new benchmark for evaluating LLM safety. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
