L$^3$: Large Lookup Layers
Researchers have introduced Large Lookup Layers (L$^3$), a novel architecture for sparse language models that aims to improve upon Mixture-of-Experts (MoE) by using static token-based routing. This approach allows models to efficiently balance memory and compute by caching information within embeddings, offering a systems-friendly design for faster training and CPU-offloaded inference. Experiments with transformers up to 2.6 billion active parameters demonstrated that L$^3$ outperforms both dense models and iso-sparse MoEs on language modeling and downstream tasks. AI
IMPACT Introduces a new architectural approach for sparse models that could improve efficiency and performance over existing MoE methods.