ReLU network analysis links Fisher information to spherical harmonics

Researchers have analyzed the Fisher information matrices of simple two-layer ReLU neural networks with random hidden weights. They found that the eigenvalue distribution concentrates significantly on specific eigenspaces, with the first three accounting for nearly all of the matrix's trace. The study identifies these dominant eigenspaces as corresponding to spherical harmonic functions of order two or less, linking this to Mercer decomposition of neural tangent kernels. AI

IMPACT Provides theoretical insights into the structure of simple neural networks, potentially informing future model design and analysis.

RANK_REASON Academic paper detailing theoretical analysis of neural network properties. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv stat.ML →

• Fisher information matrices
• Mercer decomposition
• neural tangent kernels
• spherical harmonic functions
• Yoshinari Takeishi
• ReLU networks

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →