sliding-window attention
PulseAugur coverage of sliding-window attention — every cluster mentioning sliding-window attention across labs, papers, and developer communities, ranked by signal.
2 day(s) with sentiment data
-
Xiaomi's MiMo-V2-Flash leads open-source coding benchmarks with efficient MoE architecture
Xiaomi has developed MiMo-V2-Flash, a 309-billion-parameter Mixture-of-Experts model that leads open-source options on SWE-Bench for coding tasks. This model achieves high performance with significantly less computation…
-
Xiaomi details MiMo-V2.5 AI model efficiency optimizations
Xiaomi has detailed the engineering optimizations behind its MiMo-V2.5 series of AI models, focusing on achieving efficiency for long-context reasoning and multimodal tasks. The models employ Hybrid Sliding Window Atten…
-
Evolution of Transformer Attention Mechanisms in Open-Source AI
The Transformer architecture's attention mechanism has seen significant evolution since its inception, with numerous advancements contributing to more efficient and capable large language models. Innovations like FlashA…
-
Research Rethinks Efficient Attention in Hybrid Language Models
A new research paper analyzes hybrid architectures in language models that combine full attention with efficient attention modules like sliding-window attention (SWA). The study found that efficient attention primarily …
-
Blurry Window Attention improves Transformer efficiency for long contexts
Researchers have introduced Blurry Window Attention (BLA), a novel method designed to improve the efficiency of Transformer language models in handling long contexts. BLA addresses the quadratic complexity and growing K…
-
Transformer models gain absolute position awareness from causal mask and residual stream
Researchers have identified two key architectural components in decoder-only Transformers that contribute to the model's ability to distinguish absolute position, despite positional encoding methods like RoPE primarily …
-
Open-Source LLMs Evolve: Attention, Multimodality, and Efficiency Gains
The open-source LLM landscape has seen significant shifts in recent months, with Sliding Window Attention becoming mainstream, enabling much larger context windows. QK-Norm is also gaining traction as a training stabili…