Researchers have analyzed the implicit bias of Sharpness-Aware Minimization (SAM) in improving model generalization. Their linear stability analysis reveals a quantitative relationship between SAM's perturbation radius ($\rho$), batch size ($b$), and learning rate ($\eta$), showing that these parameters influence the flatness of minima SAM seeks. Experiments on CIFAR-100 with ResNet-18 and VGG-19 models validate these findings, demonstrating that increasing $\rho$ correlates with smaller Hessian eigenvalues. The study also introduces Taylor-Locality Controlled SAM (TLC-SAM), a variant that dynamically adjusts $\rho$ to further reduce Hessian eigenvalues. AI
IMPACT Provides quantitative bounds for hyperparameter tuning in SAM, potentially improving generalization in deep learning models.
RANK_REASON Academic paper analyzing an existing optimization technique.
Read on Hugging Face Daily Papers →
AI-generated summary · Google Gemini · from 2 sources. How we write summaries →
