Researchers have developed FreqBLiMP, a new benchmark designed to evaluate the robustness of large language models (LLMs) by controlling for lexical frequency. This extension of the BLiMP benchmark specifically addresses the oversight of word frequency in existing evaluations, which is a significant factor in natural language use. FreqBLiMP regenerates minimal-pair paradigms under explicit Zipf-frequency regimes to test how grammatical preferences hold up with rare words. Initial evaluations across various LLM families show that while decreasing lexical frequency consistently lowers sentence likelihood, it only moderately impacts overall contrastive accuracy. However, this stability hides significant variations across linguistic phenomena, with LLMs performing well on morphosyntactic generalizations but degrading on tasks requiring lemma-specific information. AI
IMPACT This benchmark could lead to more robust LLMs by highlighting their fragility with rare words, potentially improving their performance in diverse real-world language scenarios.
RANK_REASON The cluster contains a research paper introducing a new benchmark for evaluating LLMs. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
