HEAPr: Hessian-based Efficient Atomic Expert Pruning in Output Space
Researchers have developed HEAPr, a new pruning algorithm designed to reduce the memory footprint of Mixture-of-Experts (MoE) large language models. Unlike previous methods that prune entire experts, HEAPr breaks down experts into smaller, atomic units. This approach utilizes second-order information from atomic expert outputs, significantly reducing computational complexity and enabling more precise compression with minimal accuracy loss. Experiments on models like DeepSeek MoE and Qwen MoE show HEAPr can achieve nearly lossless compression up to 25% pruning, reducing FLOPs by a similar margin. AI
IMPACT Enables more efficient deployment of large MoE models by reducing memory requirements without significant performance degradation.