New system optimizes MoE LLM inference with CPU-GPU collaboration

A new system called \method has been developed to improve the efficiency of Mixture-of-Experts (MoE) Large Language Models (LLMs) during inference. This system addresses the challenge of large expert memory footprints exceeding GPU capacity by intelligently assigning experts to either the CPU or GPU. \method selectively transfers high-load experts to the GPU while executing low-load experts on the CPU, dynamically adjusting assignments to minimize latency. Experiments show significant improvements in prefill speed and decode throughput compared to existing methods. AI

IMPACT Enhances inference speed and efficiency for large MoE models, potentially reducing operational costs and improving user experience.

RANK_REASON The cluster describes a new research paper detailing a novel system for optimizing AI model inference. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• central processing unit
• graphics processing unit
• Hugging Face
• \method
• Mixture of Experts (MoE)
• MoE LLMs
• PCI Express

• infra
• paper

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →