Researchers have introduced Chorus, a novel method for n-gram embeddings in language models that significantly reduces memory usage while maintaining or improving performance. Unlike previous approaches that used separate n-gram embedding tables for each layer, Chorus employs a single shared set of tables that each layer adapts through its own projection and rescaling mechanism. This shared approach allows for knowledge reuse across layers and results in substantially smaller tables, outperforming existing methods like FactorEngram and DeepSeek's Engram with a fraction of the parameters. Additionally, Chorus is compatible with memory-efficient optimizers like Ember, further reducing computational overhead. AI
IMPACT Enables more efficient scaling of language models by reducing memory requirements for n-gram embeddings.
RANK_REASON Research paper detailing a new method for n-gram embeddings in language models.
Read on Hugging Face Daily Papers →
AI-generated summary · Google Gemini · from 2 sources. How we write summaries →