Researchers have developed a counterexample to the Fourier alignment hypothesis in single-neuron modular addition. This counterexample demonstrates that a ReLU neuron can become inactive and remain frozen at a limit with Fourier energy distributed across all nonzero real frequencies. The failure occurs with positive probability under Gaussian initialization and can also happen for Clarke trajectories and smooth approximations of ReLU with fixed-step gradient descent. This finding indicates that single-frequency alignment is not a guaranteed outcome when training a single neuron on modular addition. AI
IMPACT Challenges assumptions about neural network training dynamics and potential failure modes in single-neuron operations.
RANK_REASON The item is a research paper detailing a theoretical counterexample to a specific hypothesis in neural network training. [lever_c_demoted from research: ic=1 ai=1.0]
Read on Hugging Face Daily Papers →
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
