Neural network training conserves quantities from data symmetry

Researchers have investigated whether inherent symmetries within training data can result in conserved quantities during the gradient-flow training of neural networks. Their findings suggest that for analytic and non-polynomial loss functions, data symmetries generally do not introduce additional integrals of motion. However, when using mean squared error (MSE) loss, specific scenarios involving data augmentation can lead to the emergence of extra conserved quantities. The study introduces a framework using "tensorizable networks" to describe this phenomenon, encompassing architectures like linear, polynomial, and Lightning Attention networks. AI

IMPACT This research could lead to more stable and predictable neural network training by understanding how data symmetries influence conserved quantities.

RANK_REASON The cluster contains an academic paper detailing novel research findings on neural network training. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv stat.ML →

• Data Symmetry
• Gradient-flow training
• Lightning Attention
• Mean Squared Error (MSE) loss
• Neural Networks
• Tensorizable networks

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →