New MoE inference design uses pooled HBM to cut communication latency on Ascend

Researchers have developed a new communication design for Mixture-of-Experts (MoE) inference on Ascend systems, aiming to reduce bottlenecks in token exchange. This approach eliminates intermediate relay and reordering buffers by directly placing data into destination expert windows and reading from remote ones. The system leverages globally pooled high-bandwidth memory and symmetric memory allocation, resulting in improved time to first token and competitive time per output token for MoE workloads. AI

Summary written by gemini-2.5-flash-lite from 1 source. How we write summaries →

IMPACT This research could lead to more efficient inference for large MoE models on specific hardware platforms.

RANK_REASON This is a research paper detailing a novel technical approach for optimizing MoE inference. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• Mixture-of-Experts
• MoE
• Ascend
• HBM

• paper
• infra