New method measures predictability in neural network training dynamics

Researchers have developed a new method to measure structured predictability in neural network training dynamics. This approach uses complementary probe families to analyze temporal redundancy, identifying when and under what conditions recent updates inform future parameter motion. The study found that auxiliary parameters like normalization and biases exhibit simpler dynamics compared to feature-transforming weights, which show predictable behavior in localized, time-varying pockets. This diagnostic tool was applied to vision training on CIFAR and Pythia pretraining checkpoints, revealing how architecture and training recipes influence the measured structure. AI

IMPACT Provides a new retrospective diagnostic for understanding and potentially optimizing neural network training processes.

RANK_REASON The cluster contains an academic paper detailing a new methodology for analyzing neural network training dynamics. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• alphaXiv
• CatalyzeX
• CIFAR
• DagsHub
• Gotit.pub
• Hugging Face
• IArxiv
• Influence Flower
• Pythia
• Pythia 70M
• ScienceCast

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →