New pruning technique TENP reduces MoE LLM size with minimal performance loss

Researchers have developed a new pruning technique called TENP (Trapezoidal Expert Neuron Pruning) specifically for Mixture-of-Experts (MoE) large language models. This method aims to reduce the large static parameter footprint of MoE models by selectively pruning less important experts and neurons in a structured, trapezoidal pattern. Experiments on Qwen and DeepSeek models show that TENP can achieve significant parameter reduction with minimal accuracy loss, even improving performance on code generation tasks. AI

IMPACT This technique could enable more efficient deployment of large MoE models by reducing their memory footprint.

RANK_REASON The cluster contains an academic paper detailing a new method for pruning large language models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv stat.ML →

• DeepSeek
• Mixture-of-Experts
• Qwen

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →