FrameBench benchmark tests LLM language understanding via frame semantics

Researchers have introduced FrameBench, a new benchmark designed to evaluate language understanding in large language models (LLMs) based on frame semantics. This benchmark tests whether models can implicitly enrich text with unstated information by relating lexical meaning to background knowledge, specifically by distinguishing frames evoked by the same verb in different contexts. FrameBench, constructed for both English and Japanese using FrameNet resources and human verification, presents challenges for smaller models, while several large models have demonstrated performance exceeding human reference scores. AI

IMPACT This benchmark could reveal limitations in LLM's implicit reasoning capabilities, guiding future model development.

RANK_REASON The cluster contains an academic paper introducing a new benchmark for evaluating LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• arXiv
• English
• FrameBench
• FrameNet
• Japanese

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →