A new research paper explores learning rate (LR) scaling for hybrid architectures that combine Transformer and State-Space Model (SSM) blocks, which are increasingly used in production language models. The study found that practical implementations of these hybrid models, even with simplified SSMs, exhibit near-zero LR transfer gaps across various widths and depths up to billion-parameter scale. This invariance is attributed to a combination of $\mu$P's initialization and LR scaling, along with AdamW's per-parameter normalization, which together maintain a global update-to-weight invariance and local component balance. AI
IMPACT This research could improve the training efficiency and scalability of large language models by clarifying learning rate transfer in hybrid architectures.
RANK_REASON The cluster contains a single academic paper detailing novel research findings on model architectures and training methodologies. [lever_c_demoted from research: ic=1 ai=1.0]
- AdamW
- Language Models
- Mamba
- $\mu$P
- nemotron-h
- State Space Model
- State space models: Univariate representation of a multivariate model, partial interpolation and periodic convergence
- transformer
- Zero-order hold
- Zohra
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
