Llumnix

Astrolabe system optimizes LLM serving with randomized prediction-guided scheduling

Researchers have developed Astrolabe, a novel scheduling system designed to optimize the serving of large language models (LLMs). This system employs a randomized prediction-guided approach to balance load across multip…