OrderMoE framework optimizes edge MoE model inference via expert similarity

Researchers have developed OrderMoE, a novel framework for deploying Mixture-of-Experts (MoE) models on resource-constrained edge infrastructures. OrderMoE addresses the challenges of latency and communication overhead by grouping experts based on their functional similarity. This approach aims to reduce the need for cross-server token transmission by allowing local substitute experts to be used when appropriate. Experimental results indicate that OrderMoE significantly lowers average and tail latency, decreases cross-server traffic, and reduces remote expert invocation ratios with only minor, controllable degradation in inference quality. AI

IMPACT This research could enable more efficient deployment of large language models on edge devices, improving performance and reducing communication costs.

RANK_REASON The cluster describes a new research paper detailing a novel framework for optimizing MoE model inference. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• Mixture-of-Experts
• OrderMoE

• infra
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →