Researchers have developed DP-μP, a new framework that extends Maximal Update Parametrization (μP) to differentially private optimization. Standard μP, which allows for zero-shot hyperparameter transfer in normal training, fails under DP constraints due to noise injection and gradient clipping. DP-μP addresses this by analyzing the interaction between these privacy mechanisms and feature learning, deriving a Signal-to-Noise Ratio (SNR) condition for layer-wise learning rate scaling. This framework provides specific scaling rules for DP-SGD, DP-Adam, and DP-Muon optimizers, enabling effective zero-shot transfer of learning rates across different model widths and architectures like MLPs, ViTs, and GPT-2, while reducing computational costs. AI
IMPACT Enables more efficient training of large-scale models under differential privacy, reducing computational costs and improving performance.
RANK_REASON The cluster describes a new theoretical framework and empirical validation for hyperparameter transfer in differentially private optimization, presented in an arXiv paper. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
