TrimMoE

TrimMoE framework slashes LLM inference latency by 62.8% on edge servers

Researchers have developed TrimMoE, a novel framework designed to optimize the inference of Mixture-of-Experts (MoE) large language models across distributed edge servers. This framework focuses on adaptive depth by int…