SaluNet: Enabling Total Plasticity in Normalization-Free Deep Networks
Researchers have developed SaluNet, a novel deep network architecture that eliminates the need for traditional normalization layers like BatchNorm and LayerNorm. This is achieved through a new learnable activation function called SALU, which intrinsically stabilizes signals without relying on batch statistics. SaluNet demonstrates strong performance on image classification tasks, including CIFAR-10, CIFAR-100, and ImageNet, even at very small batch sizes where normalized networks typically fail. AI
IMPACT Enables more stable and adaptable deep network training, potentially improving performance in scenarios with limited batch sizes.