Brief

A new research paper published on arXiv explores how Batch Normalization (BN) in deep neural networks can inadvertently increase the risk of data memorization and privacy breaches. The study found that BN significantly amplifies the memorization of outlier samples, making models more vulnerable to membership inference attacks. This effect is supported by both extensive empirical testing and theoretical analysis, which show BN increases the influence of outlier samples during training. AI

Researchers have introduced StableGrad, a novel optimizer-level mechanism designed to control the scale of activations and gradients in deep neural networks. This method aims to prevent training instability without relying on traditional batch normalization, which can be problematic for applications like Physics-Informed Neural Networks (PINNs). StableGrad operates by adjusting weight-gradient imbalances after backpropagation but before the optimizer update, thereby preserving the network's forward pass and physical residual accuracy. Evaluations on deep PINNs and standard architectures like ResNet and EfficientNet demonstrated StableGrad's effectiveness in improving accuracy and stabilizing optimization, even when batch normalization is removed. AI