weight-decay (WD) pulses

New research details 'canalization' phenomenon in neural network generalization

Researchers have identified a phenomenon called "canalization" in overparameterized neural networks, which describes how the selection of solutions that fit training data evolves during training. This process, observed …