Researchers have identified a phenomenon called catastrophic overfitting (CO) in fast adversarial training (FAT) of neural networks. CO occurs when robustness to multi-step attacks rapidly degrades, even as the network's performance on single-step attacks and clean data continues to improve. This happens because the training loss is largely insensitive to this weight drift. The study proposes a simple solution: using an exponential moving average (EMA) of weights, which remains robust, and resetting weights to their EMA when validation robustness drops. This method effectively eliminates CO and achieves state-of-the-art accuracy on benchmark datasets like CIFAR-100 with reduced training cost. AI
IMPACT Identifies a critical training failure mode and offers a practical, low-cost solution to improve model robustness.
RANK_REASON Academic paper detailing a novel finding and proposed solution in machine learning training. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
