LLM Inference Engines: vLLM leads on RTX 4090, llama.cpp excels on M1

A benchmark comparing three LLM inference engines—Ollama, llama.cpp, and vLLM—revealed performance differences across hardware. On a cloud RTX 4090, vLLM significantly outperformed the others, achieving 112 tokens per second compared to llama.cpp's 68 tokens per second. However, on an M1 MacBook Pro, llama.cpp took the lead with a 30% higher throughput than vLLM. Ollama consistently performed between the two on both platforms, as it acts as a wrapper around llama.cpp. AI

IMPACT Performance differences highlight hardware-specific optimizations for LLM inference, guiding deployment choices.

RANK_REASON Comparison of performance metrics for different software implementations of LLM inference. [lever_c_demoted from research: ic=1 ai=1.0]

Read on dev.to — LLM tag →

• llama.cpp
• M1
• Ollama
• RTX 4090
• vLLM

• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →