StableGrad: Backward Scale Control without Batch Normalization
Researchers have introduced StableGrad, a novel optimizer-level mechanism designed to control the scale of activations and gradients in deep neural networks. This method aims to prevent training instability without relying on traditional batch normalization, which can be problematic for applications like Physics-Informed Neural Networks (PINNs). StableGrad operates by adjusting weight-gradient imbalances after backpropagation but before the optimizer update, thereby preserving the network's forward pass and physical residual accuracy. Evaluations on deep PINNs and standard architectures like ResNet and EfficientNet demonstrated StableGrad's effectiveness in improving accuracy and stabilizing optimization, even when batch normalization is removed. AI
IMPACT Offers a new technique to stabilize deep neural network training, particularly beneficial for physics-informed models where standard normalization methods are unsuitable.