DualPace runtime boosts LLM serving throughput with GPU-CPU cooperation

Researchers have developed DualPace, a new LLM serving runtime designed to improve inference throughput and reduce latency. DualPace decouples scheduling between GPU and CPU engines, allowing the CPU to independently process requests and utilize host memory more effectively. This cooperative approach enhances serving throughput by up to 90.6% compared to existing systems like vLLM and NEO, while maintaining comparable latency. AI

IMPACT DualPace's approach could significantly improve the efficiency and cost-effectiveness of deploying large language models.

RANK_REASON The cluster describes a new research paper detailing a novel system for LLM serving. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• Advanced Matrix Extensions
• arXiv
• central processing unit
• DualPace
• graphics processing unit
• Hugging Face
• vLLM

• infra
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →