A new paper published on arXiv proposes a novel approach to evaluating large language models (LLMs) by challenging the "meaning as use" paradigm. The authors leverage Kripke's interpretation of Wittgenstein to argue that finite usage histories do not uniquely determine future rule adherence, thus questioning the adequacy of current benchmarks for establishing semantic competence. To address this, they introduce an experimental paradigm to test LLMs' capacities for abstraction and recursion, aiming to better assess their ability to preserve rules and concepts under principled domain extension. AI
IMPACT Proposes a new framework for evaluating LLM semantic competence, potentially influencing future benchmark design and model development.
RANK_REASON The cluster contains a research paper published on arXiv discussing theoretical challenges in LLM evaluation. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
