New NVLink technique speeds up Mixture-of-Experts AI inference

Researchers have developed NVLinkOneSided, a novel approach for accelerating inference in Mixture-of-Experts (MoE) models over NVLink. This method optimizes the expert parallelism all-to-all communication by enabling direct token pushing between GPUs, bypassing metadata exchange and reducing latency, particularly for small batch sizes. Benchmarks on Blackwell GPUs show NVLinkOneSided achieves 1.2-3.0x speedups over existing methods like DeepEP, making MoE layers up to 2.4x faster and competitive for decoding tasks. AI

IMPACT Optimizes MoE inference, potentially reducing latency and increasing throughput for large language models.

RANK_REASON Academic paper detailing a new inference optimization technique for MoE models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• Blackwell GPUs
• DeepGEMM
• Hugging Face
• MegaMoE
• Mixture-of-Experts
• NVLink
• NVLinkOneSided

• infra
• paper

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →