Brief

Two recent arXiv preprints explore high-dimensional ridge regression for non-identically distributed data, moving beyond standard assumptions of independent and identically distributed samples. The papers introduce variance profile models to analyze the predictive risk of ridge estimators, particularly focusing on the double descent phenomenon. Researchers used tools from random matrix theory and operator-valued free probability to derive asymptotic equivalents for risk and degrees of freedom, with numerical experiments validating their findings and highlighting how heterogeneous variance profiles can alter generalization behavior. AI

Researchers have developed a statistical framework for self-distillation in machine learning, specifically within spiked covariance models. Their analysis shows that s-step self-distillation is the optimal spectral shrinkage estimator for matrices with s spikes, outperforming existing methods. The study also highlights that s steps are necessary for this optimality and explores federated learning approaches where self-distillation remains the best local strategy. AI

A new paper investigates the role of weight decay in deep learning training stability, challenging its common perception as a simple regularization technique. The research analyzes how weight decay affects parameter dynamics and loss sharpness at the "Edge of Stability," demonstrating that it effectively slows down progressive sharpening. The study also reveals an architecture-dependent phase transition, where weight decay dampens oscillations in CNNs but stabilizes sharpness below a theoretical boundary in MLPs, driven by the alignment of parameter vectors and sharpness gradients. AI

Researchers have identified weight decay as a key parameter controlling the training regimes of transformers on modular arithmetic tasks. They introduced two new, low-cost online diagnostics—mean pairwise attention-head cosine similarity and entropy standard deviation—to monitor training dynamics from attention activations. These diagnostics, applied across various experimental conditions and model scales, effectively distinguish between memorization, generalization (grokking), and collapse, with specific transition points identified for the memorization-to-developmental boundary. AI