Neural networks can learn with flat training loss, study finds

Researchers have demonstrated that neural networks can continue to learn and improve their performance even when their training loss remains relatively flat. Using ReLU networks and specific update rules on the modular addition task, they observed that hidden features evolve while held-out accuracy increases significantly, despite a stable training cross-entropy. This phenomenon, termed 'grokking,' suggests that minimizing cross-entropy is not always a prerequisite for correct classification, as prediction confidence also influences this metric. The study further shows this coexistence of flat loss and learning in deeper networks and transformer models, indicating that a plateau in training loss does not necessarily signify an end to learning. AI

IMPACT Challenges the assumption that minimizing training loss is the sole driver of model improvement, suggesting new avenues for training more robust models.

RANK_REASON The cluster contains an academic paper detailing a novel finding in machine learning. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• alphaXiv
• CatalyzeX Code Finder for Papers
• CORE Recommender
• DagsHub
• Fourier fraction
• Gotit.pub
• grokking
• Hugging Face
• IArxiv Recommender
• Modular Addition
• Relu Networks
• ScienceCast
• supported-Polar hidden updates

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →