New ROK-FORTRESS benchmark measures LLM safety across languages and geopolitics

A new benchmark called ROK-FORTRESS has been developed to evaluate the safety of large language models (LLMs) in high-stakes national security and public safety contexts, specifically focusing on the interaction between language and geopolitical nuances. This benchmark uses the English-Korean language pair and the U.S.-South Korea geopolitical axis as a case study, employing a transcreation matrix to control for language and geopolitical grounding independently. Initial findings indicate that Korean variants of models show suppressed safety performance, and geopolitical grounding can further mitigate this suppression in some cases, suggesting that translation-only evaluations miss crucial safety alignment factors. AI

IMPACT This benchmark could lead to more robust LLM safety evaluations, particularly for multilingual applications in sensitive domains.

RANK_REASON The cluster contains a research paper detailing a new benchmark for evaluating LLM safety. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• English
• Korean
• National Security and Public Safety
• ROK-FORTRESS
• South Korea
• U.S.
• Yash Maurya

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →