Deep Linear Networks
PulseAugur coverage of Deep Linear Networks — every cluster mentioning Deep Linear Networks across labs, papers, and developer communities, ranked by signal.
3 day(s) with sentiment data
-
Adam optimizer bias linked to learning rate schedules in new research
A new research paper explores the implicit bias of the Adam optimizer in binary classification tasks, particularly its relationship with learning rate schedules. The study reveals that under certain conditions, Adam doe…
-
New research explores network depth's impact on AI training time scaling
A new paper published on arXiv explores how network depth affects training time and performance in deep linear networks. The research indicates that for tasks that are spectrally difficult, increasing network depth can …
-
New Theory Explains Orthogonalization's Effectiveness in Neural Networks
Researchers have developed a new theoretical framework to explain why orthogonalization techniques, such as Muon, are effective in optimizing neural networks. The study reveals that the Hessian of a neural network, whic…
-
New theory offers closed-form depth-width law for neural networks
Researchers have derived a closed-form depth-width law for neural collapse in deep linear networks, offering a way to understand how network architecture influences learned representations before training. This law, der…
-
Tree Tensor Networks Reveal Benign Loss Landscapes Despite Hard Targets
Researchers have explored the theoretical underpinnings of why deep neural networks, despite their complexity, often learn effectively in practice. A new study using Tree Tensor Networks (TTNs) demonstrates that even mo…
-
Grokking explained by two training clocks theory
Researchers have developed a theoretical framework to explain the phenomenon of "grokking" in machine learning, where a model fits training data and learns a generalizable rule at different rates. They propose the conce…
-
Gradient Descent with Large Step Size Redistributes Signals in Deep Networks
Researchers have demonstrated that discrete Gradient Descent with a large step size leads to a different outcome than Gradient Flow in deep linear networks with multiple pathways. While Gradient Flow predicts a "winner-…