QK-Norm
PulseAugur coverage of QK-Norm — every cluster mentioning QK-Norm across labs, papers, and developer communities, ranked by signal.
2 day(s) with sentiment data
-
Partial RoPE offers 10x memory savings in transformers with minimal performance loss
A new arXiv paper investigates the performance and convergence of partial Rotary Positional Embedding (RoPE) in transformers. The research demonstrates that applying RoPE to only a small fraction of dimensions, around 1…
-
New framework 'Neural Dimensional Analysis' improves Transformer scale consistency
Researchers have introduced Neural Dimensional Analysis (NDA), a new framework for understanding and improving the scale consistency of neural networks, particularly Transformers. NDA uses principles analogous to physic…
-
Open-Source LLMs Evolve: Attention, Multimodality, and Efficiency Gains
The open-source LLM landscape has seen significant shifts in recent months, with Sliding Window Attention becoming mainstream, enabling much larger context windows. QK-Norm is also gaining traction as a training stabili…