New Orthogonal Gradient Method Explores Noisy-Label Memorization in Neural Networks

A new research paper introduces OrthoGrad, a method that modifies the optimizer's update by removing the component of each weight gradient parallel to the current weight vector. This geometric intervention was tested on noisy-label image classification tasks, showing improved test accuracy for Convolutional Neural Networks (CNNs) on the MNIST dataset in small-data regimes by reducing the fitting of corrupted labels. While OrthoGrad can alter memorization trajectories, experiments on CIFAR-10 with ResNet-18 indicated it does not ultimately prevent noisy-label memorization, suggesting its effectiveness is dependent on the learning regime. AI

IMPACT Introduces a novel regularization technique that may offer new insights into neural network learning dynamics and memorization.

RANK_REASON Research paper introducing a novel method for neural network training. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• CIFAR-10
• CNNS
• MNIST database
• ResNet-18

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →