ExpertSparsity

New method prunes MoE language models using generic text corpora

Researchers have developed a new method called Generic TB-Coverage for pruning sparsely activated Mixture-of-Experts (MoE) language models. This technique addresses the challenge of removing redundant experts without re…