New framework analyzes neural network optimizer bias via information allocation dynamics

This paper introduces a new perspective on how neural network optimizers function, moving beyond the geometry of the final solution to examine the dynamics during training. It proposes an 'information allocation dynamics' framework that views optimizer bias as the relative distribution of training signals between weight-like and bias-like parameter pathways. This allocation can be controlled by a continuous 'preconditioning exponent p', influencing how weight and bias corrections preserve different aspects of the residual signal, thereby affecting parameter trajectories and generalization. AI

IMPACT Introduces a novel theoretical lens for understanding and potentially manipulating optimizer behavior, which could lead to improved model training and generalization.

RANK_REASON Academic paper published on arXiv detailing a new theoretical framework for understanding neural network optimization. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• arXiv
• Information Allocation Dynamics
• machine learning
• Neural Network Optimization
• Optimizer

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →