Sphere Retraction Normalizations generalize deep neural network training

Researchers have introduced Sphere Retraction Normalizations, a new framework for training deep neural networks that generalizes existing residual connection methods. This approach recasts residual connections on a Riemannian manifold, ensuring hidden states maintain a constant L2 norm by confining them to a hypersphere. The framework introduces two new methods, Proj-SpheretNorm and Cay-SpheretNorm, which utilize algebraic operations and outperform current lightweight deep connection schemes on nanoGPT, indicating that the exponential map is not always the optimal retraction for spherical residual streams. AI

IMPACT Introduces a novel framework for neural network training that generalizes existing methods and shows improved performance on benchmarks.

RANK_REASON Academic paper introducing a new normalization technique for neural networks. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• Cay-SpheretNorm
• GeoNorm
• nanoGPT
• Proj-SpheretNorm
• p-SpheretNorm
• Riemannian manifold

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →