Analyzing Stream Collapse in Hyper-Connections: From Diagnosis to Mitigation
A new research paper explores the phenomenon of "stream collapse" in Hyper-Connections (HC) models, which utilize multiple residual streams instead of a single one. The study found that these models often exhibit dominant-stream usage, with information and features concentrating in one stream, limiting the intended multi-stream information exchange. Researchers demonstrated that breaking the initial symmetry among streams can reduce this dominance and improve model performance. AI
IMPACT Identifies a performance bottleneck in multi-stream Transformer architectures, suggesting methods to improve efficiency and specialization.