LLM inference tools vLLM, llama.cpp, Ollama benchmarked on VRAM limits

A benchmark comparison of vLLM, llama.cpp, and Ollama reveals significant differences in performance, particularly when dealing with large language models that exceed the available VRAM. While vLLM excels in throughput within 24GB of VRAM, achieving up to 5.4x scaling with increased concurrency, it fails entirely when models require more than approximately 22GB. In contrast, llama.cpp and Ollama can handle these larger models by spilling to system RAM, albeit at a much slower single-digit token-per-second rate. Notably, llama.cpp demonstrates a substantial advantage in time-to-first-token when manually offloading layers compared to Ollama's automatic approach. AI

IMPACT Highlights performance differences in LLM inference tools, guiding users on optimal choices based on hardware constraints and model size.

RANK_REASON The item benchmarks and compares different software tools for running large language models, focusing on performance characteristics and limitations. [lever_c_demoted from research: ic=1 ai=1.0]

Read on dev.to — LLM tag →

• Gemma~3
• Gemma 4
• GLM-4.5-Air
• GPT-OSS 120B
• llama.cpp
• Ollama
• RTX 3090
• vLLM

• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →