Researchers have developed a new framework called Hyperparameter Transfer Laws to better understand and predict how hyperparameters should be adjusted when scaling neural network architectures. This framework introduces the concept of 'effective depth' to unify various multi-path networks like CNNs, ResNets, and Transformers. Experiments show that optimal learning rates decay with effective depth following a universal -3/2 power law, enabling reliable zero-shot transfer of learning rates across different depths and widths. AI
IMPACT This research could significantly reduce the computational cost of training deep learning models by enabling more effective hyperparameter transfer.
RANK_REASON The cluster contains a research paper detailing a new theoretical framework and experimental validation for hyperparameter transfer in neural networks. [lever_c_demoted from research: ic=1 ai=1.0]
- alphaXiv
- arXiv
- CatalyzeX
- CNNS
- DagsHub
- Gotit.pub
- Hugging Face
- IArxiv
- Maximal Update Parametrization
- Resnet
- ScienceCast
- Shenxi Wu
- transformers
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
