SliceScheduler boosts GPU utilization for multi-tenant LLM serving

Researchers have developed SliceScheduler, a novel dynamic operator-level scheduling system designed to enhance efficiency in multi-tenant model serving. This system utilizes a Global Mapping Graph to provide a real-time, cluster-wide view of operator dependencies and execution states. By employing a global simulator, SliceScheduler predicts operator execution and memory evolution, enabling it to exploit fragmented idle slices while adhering to service-level agreements and avoiding memory violations. Implemented as a PyTorch backend, SliceScheduler has demonstrated significant improvements in token throughput, achieving up to a 2.29x increase over existing methods with minimal SLA violations. AI

IMPACT Improves GPU utilization and token throughput for large language model serving infrastructure.

RANK_REASON Academic paper detailing a new system for model serving efficiency. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• Global Mapping Graph
• graphics processing unit
• PyTorch
• service-level agreement
• SliceScheduler

• infra
• paper

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →