Researchers have developed a novel technique called Lookahead Expert Scheduling (LES) to enable large Mixture-of-Experts (MoE) models to run efficiently on mobile devices. This method addresses the bottleneck of loading expert weights from storage by predicting future expert needs and preloading them during computation. The implementation, demonstrated with a 35B MoE model on an iPhone 18 Pro Max, achieved a speed of 20.51 tokens per second, a significant improvement over baseline performance, while maintaining model accuracy and minimizing memory usage. The open-source implementation suggests this approach could be applicable to other MoE models. AI
IMPACT This development could significantly lower the barrier for running advanced AI models on edge devices, enabling new mobile applications and offline AI capabilities.
RANK_REASON The item describes a novel technique for running large MoE models on consumer hardware, detailing performance improvements and implementation specifics. [lever_c_demoted from research: ic=1 ai=1.0]
- 35B MoE
- DSpark
- iPhone
- iPhone 18 Pro Max
- llama.cpp-omni
- Lookahead Expert Scheduling (LES)
- MiniCPM-V 4.7
- OpenBMB
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
