When Similar Means Different: Evaluating LLMs on Arabic--Hebrew Cognates
Researchers have developed SemCog Bench, a new benchmark designed to evaluate how well large language models (LLMs) handle cognates between Arabic and Hebrew. The benchmark includes 1,858 word pairs and sentence-level annotations to test identification and semantic disambiguation. Evaluations revealed that LLMs perform well on true cognates but struggle significantly with false friends and loanwords, indicating a reliance on surface-level similarity rather than deep semantic understanding. Even with contextual cues, performance gains were modest, highlighting a fundamental limitation in current LLMs for resolving cross-lingual meaning conflicts. AI
IMPACT Highlights limitations in LLM cross-lingual understanding, potentially guiding future model development for nuanced semantic reasoning.