A new language model called EntropyBeam has demonstrated superior performance on the nanoGPT Shakespeare benchmark, achieving lower cross-entropy than the nanoGPT model. EntropyBeam operates without trainable parameters and completes its learning in a single pass by computing count tables that map character contexts to next-character frequencies. While it stores more data (2.7 million context-transition entries) compared to nanoGPT's learned parameters (60,192), its unique approach of combining multiple weighted orders through a weighted geometric mean results in higher accuracy. AI
IMPACT Introduces a novel, parameter-free approach to language modeling that achieves competitive results on character-level tasks.
RANK_REASON The item describes a new language model and its performance on a specific benchmark, fitting the research category. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
