Brief

Researchers have introduced Olmo Hybrid, a new 7-billion parameter language model that combines recurrence and attention mechanisms. This hybrid architecture, featuring Gated DeltaNet layers, demonstrates superior performance and more efficient scaling compared to traditional transformers and its predecessor, Olmo 3. The study theoretically and practically shows that Olmo Hybrid can express tasks beyond both pure transformers and linear RNNs, including code execution, suggesting a promising new direction for language model development. AI

Researchers have explored linear RNNs (LRNNs) as language models, noting their expressivity and parallelizability. A new paper connects LRNNs to arithmetic circuits, explaining their parallel nature by showing they are similar to log-depth circuits, unlike nonlinear RNNs which can solve more complex problems. This theoretical work identifies expressivity differences between LRNN variants and provides a foundation for designing LLM architectures that balance expressivity and parallelism. AI