Researchers have developed a new method called Generic TB-Coverage for pruning sparsely activated Mixture-of-Experts (MoE) language models. This approach uses generic text corpora like WikiText2 and C4 for calibration, unlike existing methods that rely on a single aggregated importance score. By profiling per-expert utility separately on each corpus and enforcing a budget-based coverage rule, Generic TB-Coverage preserves high-utility experts before creating the final pruning mask. Experiments on Qwen1.5-MoE-A2.7B and DeepSeek-MoE-16B-Base models showed improved average accuracy and reduced perplexity degradation compared to other pruning methods, particularly under aggressive pruning scenarios. AI
IMPACT This method could enable more efficient deployment of large MoE models by reducing their size without significant performance loss, especially when downstream calibration data is unavailable.
RANK_REASON The cluster describes a new method proposed in a research paper for pruning MoE language models. [lever_c_demoted from research: ic=1 ai=1.0]
Read on Hugging Face Daily Papers →
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
