New criterion optimizes weight initialization for deep neural networks

Researchers have developed a new mathematical criterion for selecting the optimal variance of weight initialization in deep neural networks (DNNs). By analyzing stochastic gradient descent (SGD) through a continuous-time Fokker-Planck equation, they established a connection between the initial distribution and the expected cost function via Kullback-Leibler divergence. This method provides a concrete, non-heuristic way to determine initialization scale, leading to lower training loss and higher test accuracy compared to conventional He-normal initialization, as validated on MNIST and Fashion-MNIST datasets. AI

IMPACT Provides a mathematically grounded method to improve DNN training efficiency and performance.

RANK_REASON Academic paper detailing a new theoretical criterion for optimizing DNN initialization. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• deep neural network
• Fashion-MNIST
• He-normal initialization
• Hiroshi Horii
• Kullback--Leibler divergence
• MNIST database
• stochastic gradient descent

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →