FusionML boosts Apple Silicon AI inference with CPU-GPU co-execution

Researchers have developed FusionML, a system that optimizes transformer inference on Apple Silicon by co-executing CPU and GPU operations. By addressing limitations in MLX's scheduler that caused serialization, FusionML enables a row-split approach for transformer prefill, leading to speedups of 1.15x to 1.38x for Llama-shaped models and 1.18x to 1.25x faster time-to-first-token for Qwen2.5-7B. The system's effectiveness is characterized across multiple Apple Silicon generations, with findings indicating that decode operations do not benefit, and certain training scenarios or the Apple Neural Engine introduce prohibitive overheads. AI

IMPACT Optimizes AI inference on Apple Silicon, potentially improving performance for on-device AI applications.

RANK_REASON Paper detailing a new method for optimizing AI model inference on specific hardware. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• Apple Neural Engine
• Apple Silicon
• central processing unit
• graphics processing unit
• llama
• Mlx
• mlx-lm
• qwen2.5:7b

• infra
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →