TEMPO dispatcher optimizes MoE serving by accounting for memory and compute regimes

Researchers have developed TEMPO, a novel makespan-aware dispatcher designed to optimize expert-parallel (EP) Mixture-of-Experts (MoE) model serving. Unlike previous methods that assume linear scaling of expert time, TEMPO accounts for two distinct regimes: memory-bound (HBM weight streaming) and compute-bound (grouped GEMM rounds). The system formalizes per-batch dispatch as a fixed-charge makespan problem, solving it efficiently to improve throughput and reduce latency, particularly in scenarios where both regimes are present simultaneously. AI

IMPACT Optimizes MoE model serving efficiency, potentially leading to faster inference and reduced costs for large language models.

RANK_REASON This is a research paper detailing a new method for optimizing AI model serving. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• DeepSeek-V3
• Qwen3-235B
• TEMPO

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →