cMoLLM

New cMoLLM architecture scales LLMs via dynamic convolutions

Researchers have introduced cMoLLM, a novel approach to scaling large language models by incorporating a mixture-of-experts (MoE) style throughout the entire model pipeline, rather than just in the feed-forward networks…