DivMoE

DivMoE framework enables efficient fine-grained MoE model upcycling

Researchers have developed DivMoE, a novel framework for efficiently creating Mixture-of-Experts (MoE) models from pre-trained dense models. Previous methods struggled with fine-grained upcycling, leading to accuracy dr…