Hyper-Connections
PulseAugur coverage of Hyper-Connections — every cluster mentioning Hyper-Connections across labs, papers, and developer communities, ranked by signal.
1 day(s) with sentiment data
-
New Spectral-Sphere-Constrained Hyper-Connections Improve AI Model Stability
Researchers have introduced Spectral-Sphere-Constrained Hyper-Connections ($\mathrm{s}^2$HC) to address limitations in existing Hyper-Connection (HC) models. Current methods like Manifold-Constrained Hyper-Connections (…
-
New Orthogonal Hyper-Connections improve Transformer stability
Researchers have introduced Orthogonal Hyper-Connections (oHC) as a novel architecture for Transformers, aiming to improve training stability and performance. Unlike standard Hyper-Connections (HC) or Manifold-Constrain…
-
xHC method expands transformer streams beyond N=4 for improved LLM pre-training
Researchers have introduced xHC (Expanded Hyper-Connections), a novel method for scaling transformer models beyond the typical limit of N=4 streams. This new approach addresses bottlenecks in previous Hyper-Connections …
-
New paper proposes 'Hyper-Connections' to improve transformer architectures
A new paper introduces the "y=x Problem" and proposes "Hyper-Connections" as a method to improve transformer architectures. This approach aims to dynamically route information within neural networks, moving beyond stati…
-
Hyper-Connections models suffer from stream collapse, researchers find
A new research paper explores the phenomenon of "stream collapse" in Hyper-Connections (HC) models, which utilize multiple residual streams instead of a single one. The study found that these models often exhibit domina…
-
KromHC improves neural network training with Kronecker products
Researchers have introduced KromHC, a novel method for improving neural network training stability and scalability. KromHC addresses limitations in existing Hyper-Connections (HC) by using Kronecker products of smaller …