New ML-based GPU caching algorithm LCR boosts LLM inference speed

Researchers have developed a new GPU caching algorithm called Learning-Augmented LRU (LALRU) designed to improve efficiency during AI inference. This algorithm integrates learned predictions with caching policies to ensure both near-optimality with accurate predictions and bounded performance degradation with inaccurate ones. A practical implementation named LCR, built upon LALRU, demonstrated significant improvements in LLM workloads, reducing P99 time-to-first-token by up to 28.3%, and boosting throughput by up to 24.2% for DLRM workloads. AI

IMPACT Improves inference efficiency and throughput for LLM and DLRM workloads, potentially lowering operational costs.

RANK_REASON Academic paper introducing a new algorithm for GPU caching in AI inference.

Read on arXiv cs.LG →

• DLRM
• GPU
• LALRU
• LLM
• LRU

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →