KromHC: Manifold-Constrained Hyper-Connections with Kronecker-Product Residual Matrices
Researchers have introduced KromHC, a novel method for improving neural network training stability and scalability. KromHC addresses limitations in existing Hyper-Connections (HC) by using Kronecker products of smaller doubly stochastic matrices to parameterize the residual matrix. This approach guarantees exact double stochasticity while significantly reducing the number of trainable parameters compared to previous methods. Experiments demonstrate that KromHC performs comparably to or better than state-of-the-art variants with substantially fewer parameters. AI
IMPACT Introduces a more efficient and stable method for training neural networks, potentially improving performance and reducing computational costs.