LLM serving scheduler extended to support multi-tier SLAs

Researchers have extended the Llumnix LLM serving scheduler to support more than two priority tiers, enabling it to better manage heterogeneous service-level objectives (SLOs). The enhanced scheduler was evaluated using the Vidur simulator and demonstrated significant improvements in cost-effectiveness and latency reduction across multiple priority levels. Experiments showed that four priority tiers offered the best trade-off, achieving substantial speedups and cost-per-latency improvements compared to baseline systems like INFaaS, vLLM, Orca, and Sarathi-Serve. AI

IMPACT Enhances LLM serving efficiency and cost-effectiveness, potentially improving user experience for diverse applications.

RANK_REASON Academic paper detailing an extension to an LLM serving scheduler. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• arXiv
• Hugging Face
• INFaaS
• Large Language Model Serving
• Llumnix
• Orca
• Sarathi-Serve
• vLLM

• infra
• paper

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →