Hyper-Connections
PulseAugur coverage of Hyper-Connections — every cluster mentioning Hyper-Connections across labs, papers, and developer communities, ranked by signal.
-
xHC method expands transformer streams beyond N=4 for improved LLM pre-training
Researchers have introduced xHC (Expanded Hyper-Connections), a novel method for scaling transformer models beyond the typical limit of N=4 streams. This new approach addresses bottlenecks in previous Hyper-Connections …
-
New paper proposes 'Hyper-Connections' to improve transformer architectures
A new paper introduces the "y=x Problem" and proposes "Hyper-Connections" as a method to improve transformer architectures. This approach aims to dynamically route information within neural networks, moving beyond stati…
-
Hyper-Connections models suffer from stream collapse, researchers find
A new research paper explores the phenomenon of "stream collapse" in Hyper-Connections (HC) models, which utilize multiple residual streams instead of a single one. The study found that these models often exhibit domina…
-
KromHC improves neural network training with Kronecker products
Researchers have introduced KromHC, a novel method for improving neural network training stability and scalability. KromHC addresses limitations in existing Hyper-Connections (HC) by using Kronecker products of smaller …