Conservation Laws from Data Symmetry in Neural Networks
Researchers have investigated whether inherent symmetries in training data can result in conserved quantities during the gradient-flow training of neural networks. Their findings indicate that for analytic and non-polynomial loss functions, data symmetries generally do not introduce additional integrals of motion. However, with mean squared error loss, specific data augmentation techniques can lead to the emergence of conserved quantities. The study introduces a framework using 'tensorizable networks' to model this phenomenon, encompassing architectures like linear, polynomial networks, and Lightning Attention. AI
IMPACT This research could lead to more stable and predictable neural network training by identifying conserved quantities, potentially improving model performance and understanding.