New benchmark reveals language-dependent risks in multilingual LLM instruction compliance

A new benchmark called XIH-Bench has been developed to evaluate instruction hierarchy compliance in multilingual large language models. The benchmark reveals that compliance varies significantly across languages, with a phenomenon termed the 'Language Boundary Effect' showing that cross-language instruction conflicts result in higher compliance than same-language conflicts. This language-dependent asymmetry and the potential for lower-priority instructions in model-favored languages to be harder to override present reliability and security risks in multilingual LLM deployments. AI

IMPACT Highlights potential security and reliability risks in multilingual LLM applications due to language-dependent instruction compliance.

RANK_REASON The cluster contains a research paper introducing a new benchmark for evaluating multilingual LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• alphaXiv
• arXiv
• CatalyzeX
• DagsHub
• English
• Gotit.pub
• Hugging Face
• ScienceCast
• XIH-Bench

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →