vLLM 0.31.0 speeds up model restarts with GPU weight caching

vLLM has released version 0.31.0, introducing a "preload" feature designed to speed up engine restarts by keeping quantized model weights resident in GPU memory. This feature, along with experimental CRIU-based snapshots, aims to reduce the time spent loading weights from disk during engine restarts, which can significantly improve baseline utilization and handle traffic bursts more effectively. However, the preload daemon is node-local and still requires an initial disk load, and a known issue exists where the weight cache fingerprint does not verify the actual weight values, potentially leading to serving incorrect models. AI

IMPACT Improves inference server efficiency by reducing model load times, potentially enabling higher GPU utilization and faster scaling.

RANK_REASON This is a software infrastructure update for an open-source project, not a frontier release from a major AI lab.

Read on dev.to — LLM tag →

• Criu
• CUDA IPC
• meta-llama/llama-3.1-8b-instruct
• MetaX N300-A
• Qwen3_8B
• vLLM
• vLLM 0.31.0
• vLLM-IPC-Cache 0.2.0
• vLLM preload

• infra
• product

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →