Beyond $\ell_2$-norm and $\ell_\infty$-norm: A Curvature-Inspired $\ell_p$-Norm Scheme for Deep Neural Networks
Researchers have introduced a new optimization scheme for deep neural networks that moves beyond the limitations of existing $\ell_2$ and $\ell_\infty$ norms. This novel $\ell_p$-norm scheme dynamically adjusts the value of $p$ during training, initially using a large $p$ to manage high-curvature directions and then gradually decreasing $p$ towards 2 for more stable convergence. Theoretical analysis suggests this approach achieves an $O(T^{-1/2})$ convergence rate in non-convex settings, and experiments on datasets like CIFAR and ImageNet with various neural networks demonstrate its effectiveness. AI
IMPACT Introduces a novel optimization technique that could improve training efficiency and generalization performance for deep learning models.