Symbolic regression discovers novel neural network optimizers

Researchers have explored the use of symbolic regression to discover novel weight-update rules for feed-forward neural networks. In experiments across 30 benchmark and neural network combinations, the symbolic regression approach yielded update rules that outperformed established optimizers in 25 instances, achieving an average reduction in Mean Squared Error (MSE) of 44.47%. The newly found rules often incorporate adaptive normalization, momentum-like elements, and nonlinear transformations, suggesting symbolic regression's potential for creating efficient optimizer variants, though larger-scale validation is recommended. AI

IMPACT Suggests a new avenue for developing more efficient neural network optimizers, potentially improving training performance.

RANK_REASON Academic paper detailing a novel method for discovering neural network optimizers. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• arXiv
• feedforward neural network
• gradient
• Hugging Face
• Moment Estimates Derived from Poincaré and Logarithmic Sobolev Inequalities
• Momentum
• Symbolic regression

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →