QoS

DriftSched improves LLM inference efficiency with adaptive scheduling

Researchers have developed DriftSched, a framework to improve the efficiency of multi-tenant GPU inference for large language models. This system addresses the challenge of runtime token drift, where actual output lengt…