SlimPajama-6B

SpecDrop introduces parameter-free routing for specialized AI models

Researchers have introduced SpecDrop, a novel parameter-free routing method for Mixture of Experts (MoE) models that leverages category labels for specialization. Unlike traditional MoE approaches that rely on learned r…