Ubiquity of Emergent Hebbian Dynamics in Regularized Learning
A new research paper explores how regularization techniques in machine learning can lead to emergent Hebbian dynamics. The study demonstrates that L2 weight decay, a common regularization method, can cause the learning signal within update rules to align with a Hebbian direction. This phenomenon is not limited to specific learning algorithms and can occur even before learning has concluded. Furthermore, the research suggests that stochastic noise in the learning signal can induce anti-Hebbian alignment, complicating the interpretation of synaptic measurements and necessitating experiments to differentiate between true Hebbian computation and these emergent signatures. AI
IMPACT Suggests that observed Hebbian-like structures in neural networks may not always stem from explicit Hebbian learning mechanisms.