How Does the ReLU Activation Affect the Implicit Bias of Gradient Descent on High-dimensional Neural Network Regression?
A new research paper explores how the ReLU activation function influences the implicit bias of gradient descent in high-dimensional neural network regression. The study, using a novel primal-dual analysis, demonstrates that for sufficiently high-dimensional random data, the implicit bias approximates the minimum $\ell_2$-norm solution. This approximation is achieved with high probability, with a gap on the order of $\Theta(\sqrt{n/||\lambda||_1})$, where $n$ is the number of training examples and $\lambda$ represents the spectrum of the data covariance matrix. The findings indicate that the ReLU activation pattern quickly stabilizes under these conditions. AI
IMPACT Provides theoretical insights into the behavior of gradient descent with ReLU activations in overparameterized models.