gated linear attention
PulseAugur coverage of gated linear attention — every cluster mentioning gated linear attention across labs, papers, and developer communities, ranked by signal.
2 day(s) with sentiment data
-
New DAMI method enhances long-context retrieval in gated linear attention models
Researchers have developed a new method called Dual-Axis Memory Initialization (DAMI) to improve long-context retrieval in gated linear attention models. DAMI configures key parameters before training to ensure better m…
-
i14's Laplace 3B linear attention model trained and served on AWS Trainium2
i14 has developed Laplace 3B, a linear attention model that maintains a fixed-size recurrent state, unlike traditional transformer models whose key-value cache grows with context length. This innovation allows for more …
-
New research explores attention mechanisms, sparse models, and LLM monitoring
Multiple research papers explore advancements in attention mechanisms for large language models and neural networks. One paper details a fix for the llama.cpp repository to address KQ bias in attention calculations. Oth…