New DeaMoE architecture boosts LLM decoding efficiency

Researchers have introduced DeaMoE, a novel Mixture of Experts (MoE) architecture designed to enhance decoding efficiency for large language models, particularly in small-batch scenarios. This new structure groups experts into departments, allowing for parameter sharing within departments while retaining unique parameters for individual experts. DeaMoE employs a two-stage routing strategy to minimize redundant expert loading, leading to significant improvements in decoding speed and reduced memory usage. Experiments show DeaMoE can reduce loaded weights by up to 50.9% and achieve speedups of up to 2.00x on NVIDIA H100 GPUs. AI

IMPACT This architecture could significantly speed up real-time AI applications like coding assistants by improving LLM decoding efficiency.

RANK_REASON Academic paper introducing a new model architecture. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• DeaMoE
• DeepSeek-V3
• Mixture of Experts (MoE)
• NVIDIA H100

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →