Diagonal Linear Networks
PulseAugur coverage of Diagonal Linear Networks — every cluster mentioning Diagonal Linear Networks across labs, papers, and developer communities, ranked by signal.
1 day(s) with sentiment data
-
New L2 regularization variant enhances continual learning in linear networks
Researchers have developed a new weighted variant of L2 regularization designed to improve continual learning in diagonal linear networks. This method aims to preserve the sparsity bias that L2 regularization promotes i…
-
New Theory Explains "Grokking" in Machine Learning Models
Researchers have developed a statistical theory to explain the phenomenon of "grokking," where a model learns underlying signals at a different stage than it fits training data. The theory characterizes how regularizati…
-
Paper analyzes SAM's impact on diagonal linear network training
This paper analyzes the training dynamics of diagonal linear networks using Stochastic Sharpness-Aware Minimization (SAM). Researchers demonstrate that noise introduced during training acts as a stochastic form of SAM, …
-
Paper analyzes SGD dynamics in high-dimensional linear networks
A new paper details the high-dimensional behavior of stochastic gradient descent (SGD) on diagonal linear networks. The research shows that in high dimensions, SGD dynamics can be accurately modeled by a stochastic diff…