Brief

Researchers have developed HEAPr, a new pruning algorithm designed to reduce the memory footprint of Mixture-of-Experts (MoE) large language models. Unlike previous methods that prune entire experts, HEAPr breaks down experts into smaller, atomic units. This approach utilizes second-order information from atomic expert outputs, significantly reducing computational complexity and enabling more precise compression with minimal accuracy loss. Experiments on models like DeepSeek MoE and Qwen MoE show HEAPr can achieve nearly lossless compression up to 25% pruning, reducing FLOPs by a similar margin. AI

Researchers have introduced ROMER, a post-training calibration framework designed to enhance the robustness of Mixture-of-Experts (MoE) Large Language Models (LLMs) when deployed on analog Compute-in-Memory (CIM) systems. This framework addresses hardware imperfections in CIM by replacing underutilized experts and recalibrating router decisions to maintain load balance and optimal routing under noisy conditions. Experiments show ROMER significantly reduces perplexity for models like DeepSeek-MoE, Qwen-MoE, and OLMoE when subjected to real-chip noise. AI