A new study, Rel-Bloom, evaluates multilingual LLMs on their ability to reason over relational knowledge provided in prompts, rather than relying on memorized associations. The research found that while models like Gemini perform well across most tasks, accuracy drops significantly at higher reasoning levels (Analyze and Evaluate) for open-weight models. Difficulty rankings for relations and items remained consistent across languages, suggesting a shared underlying structure. The study also indicated that counterfactual contexts are generally harder, and removing entity identities can sometimes improve performance in certain model regimes, highlighting that greater capability does not automatically ensure reliable relational reasoning. AI
IMPACT Highlights limitations in current LLMs' relational reasoning, suggesting areas for future model development and evaluation.
RANK_REASON Academic paper detailing a new benchmark and its findings on LLM reasoning capabilities. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
