Ollama outperforms vLLM in speed and cost for Qwen2.5-7B-Instruct on A10 GPU

A recent benchmark test on a 24 GB NVIDIA A10 GPU compared Ollama with a Q4_K_M quantized model against vLLM using bfloat16 precision for the Qwen2.5-7B-Instruct model. The results indicated that Ollama was faster and more cost-effective per token across various user loads, from 1 to 16 concurrent users. While Ollama achieved higher throughput, vLLM maintained a slight edge in time-to-first-token when multiple users were requesting responses simultaneously. In terms of accuracy on the GSM8K benchmark, Ollama's Q4_K_M model scored 90.0%, slightly lower than vLLM's 91.2% with bfloat16. AI

IMPACT Ollama's efficiency may encourage wider adoption for single-GPU LLM deployments.

RANK_REASON Benchmark comparison of two LLM serving frameworks. [lever_c_demoted from research: ic=1 ai=0.7]

Read on dev.to — LLM tag →

• bfloat16
• GSM8K
• llama.cpp
• Nvidia
• Ollama
• Q4_K_M
• Qwen2.5-7B-Instruct
• Runcrate
• vLLM

• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →