KromHC improves neural network training with Kronecker products

Researchers have introduced KromHC, a novel method for improving neural network training stability and scalability. KromHC addresses limitations in existing Hyper-Connections (HC) by using Kronecker products of smaller doubly stochastic matrices to parameterize the residual matrix. This approach guarantees exact double stochasticity while significantly reducing the number of trainable parameters compared to previous methods. Experiments demonstrate that KromHC performs comparably to or better than state-of-the-art variants with substantially fewer parameters. AI

IMPACT Introduces a more efficient and stable method for training neural networks, potentially improving performance and reducing computational costs.

RANK_REASON This is a research paper detailing a new method for neural network parameterization. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• Birkhoff polytope
• Hyper-Connections
• KromHC
• Manifold-Constrained Hyper-Connections
• Sinkhorn-Knopp algorithm
• Wuyang Zhou

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →