New ALTO recipe enables efficient MXFP4 training for MoE LLMs

Researchers have developed a new training recipe called ALTO for the GPT-OSS-20B mixture-of-experts model, enabling it to train effectively using the MXFP4 sub-8-bit arithmetic format. This approach combines existing techniques with a simplified de-oscillation scheme, significantly reducing the convergence overhead compared to standard BF16 training. The ALTO recipe successfully closed a substantial portion of the quantization error gap, achieving a validation loss very close to the BF16 baseline and drastically cutting the convergence time needed to meet quality targets. AI

IMPACT Enables more efficient training of large language models by reducing memory and compute requirements.

RANK_REASON Academic paper detailing a new training methodology for LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• Abecassis, Felix et al.
• AMD MI300
• arXiv:2509.25149
• arXiv:2510.27527
• bfloat16
• C4 model
• GPT OSS 20B
• MLPerf Small MoE Training
• MXFP4
• TetraJet-v2

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →