New DTop-p MoE offers dynamic routing for efficient foundation model training

Researchers have introduced DTop-p MoE, a novel routing mechanism for sparse Mixture-of-Experts (MoE) architectures used in foundation model pre-training. This method dynamically adjusts the Top-p probability threshold using a Proportional-Integral controller and layer-wise expert selection under a global sparsity constraint. Experiments show DTop-p MoE outperforms standard Top-k and fixed Top-p methods in Large Language Models and Diffusion Transformers, while maintaining comparable computational costs. AI

IMPACT Introduces a more efficient routing mechanism for MoE architectures, potentially improving training scalability and performance for large models.

RANK_REASON The cluster contains a research paper detailing a new method for foundation model pre-training. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• Can Jin
• Diffusion Transformers
• DTop-p MoE
• Large Language Models
• Mixture-of-Experts

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →