Researchers have developed Astrolabe, a novel scheduling system designed to optimize the serving of large language models (LLMs). This system employs a randomized prediction-guided approach to balance load across multiple LLM instances without the need for costly migration-based rebalancing. Astrolabe combines response-length estimation, latency prediction, and a power-of-two-choices dispatch policy to enhance load balancing and reduce latency. Experiments show Astrolabe can match or exceed the performance of existing baselines, significantly reducing mean and P99 time-to-first-token and end-to-end latency, while also decreasing predictor CPU usage. AI
IMPACT Optimizes LLM serving infrastructure, potentially reducing costs and improving response times for AI applications.
RANK_REASON The cluster contains a research paper detailing a new scheduling system for LLM serving. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
