New pruning methods reduce MoE LLM memory footprint while preserving performance

Researchers have developed new methods for pruning experts in Mixture-of-Experts (MoE) large language models, addressing the significant memory overhead associated with storing all experts. The proposed techniques, Projection Energy Expert Pruning (PEEP) and Distributional Coverage Expert Pruning (DCEP), aim to preserve the structure and functional subspace of expert outputs. Experiments on DeepSeek-V2-Lite and Qwen3-30B-A3B models show that these methods outperform existing baselines in maintaining performance across various benchmarks, even when retaining only 50% or 75% of the experts. AI

IMPACT These pruning techniques could significantly reduce the computational and memory costs of deploying large language models, making them more accessible.

RANK_REASON The cluster contains an academic paper detailing novel methods for optimizing large language models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• DeepSeek-V2-Lite
• Distributional Coverage Expert Pruning
• mixture of experts
• Projection Energy Expert Pruning
• Qwen3-30B-A3B

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →