Astrolabe system optimizes LLM serving with randomized prediction-guided scheduling

Researchers have developed Astrolabe, a novel scheduling system designed to optimize the serving of large language models (LLMs). This system employs a randomized prediction-guided approach to balance load across multiple LLM instances without the need for costly migration-based rebalancing. Astrolabe combines response-length estimation, latency prediction, and a power-of-two-choices dispatch policy to enhance load balancing and reduce latency. Experiments show Astrolabe can match or exceed the performance of existing baselines, significantly reducing mean and P99 time-to-first-token and end-to-end latency, while also decreasing predictor CPU usage. AI

IMPACT Optimizes LLM serving infrastructure, potentially reducing costs and improving response times for AI applications.

RANK_REASON The cluster contains a research paper detailing a new scheduling system for LLM serving. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• A100 GPUs
• arXiv
• astrolabe
• LLaMA-2 7B
• LLM
• Llumnix
• Qwen2-7B
• ShareGPT
• Wei-Dan Jiang

• infra
• paper

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →