Prime Intellect has enhanced DeepSeek's Mega MoE kernel by adding a fused, overlapping backward pass, making it suitable for training Mixture-of-Experts (MoE) models. This new kernel, Prime Mega MoE, significantly speeds up training steps by up to 34% compared to existing baselines and up to 55% when compared against TorchTitan. The performance gains are more pronounced in the forward pass, reaching up to 3.11x, while backward pass improvements are more modest, around 1.24x. The kernel supports various NVIDIA GPUs, including Hopper and Blackwell architectures, and is available as a dispatch backend within the prime-rl trainer. AI
IMPACT Accelerates training for Mixture-of-Experts models, reducing GPU compute costs and time.
RANK_REASON This is an optimization kernel for training existing MoE models, not a new frontier model release.
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
