SmartMixed: A Two-Phase Training Strategy for Adaptive Activation Function Learning in Neural Networks
Researchers have developed SmartMixed, a new two-phase training strategy that enables neural networks to learn optimal activation functions for individual neurons. The first phase uses a differentiable mixture mechanism for neurons to select from a pool of candidate functions, while the second phase fixes these selections for computational efficiency. Experiments on the MNIST dataset with feedforward networks showed that neurons in different layers develop distinct activation function preferences, outperforming models with a single fixed activation function. AI
IMPACT Enables more efficient and potentially more powerful neural network architectures by optimizing activation functions at a granular level.