New ThunderEP design boosts MoE inference on consumer GPUs

Researchers have developed ThunderEP, a new communication design for efficiently running large Mixture-of-Experts (MoE) models on consumer GPUs connected via PCIe. This system addresses the communication bottlenecks inherent in these setups, where data transfers typically involve the CPU. ThunderEP aims to bypass CPU relay hops and utilize DMA engines to avoid contention with computational resources, thereby reducing synchronization latency. Evaluations on systems with RTX 4090 and RTX 5090 GPUs demonstrate significant speedups over existing methods like NCCL, achieving up to 1.66x end-to-end improvement for MoE inference. AI

IMPACT Optimizes inference for large MoE models on consumer hardware, potentially lowering barriers for researchers and developers.

RANK_REASON The item is a research paper detailing a new technical approach for optimizing AI model inference on specific hardware. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• Chris Jaehwan Lee
• graphics processing unit
• Mixture of Experts (MoE)
• NCCL
• PCI Express
• RTX 4090
• RTX 5090
• ThunderEP
• vLLM

• infra
• paper

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →