User benchmarks show mainline llama.cpp outperforms ik_llama.cpp in multi-GPU setups

A user on r/LocalLLaMA is questioning the reported speed advantage of the ik_llama.cpp fork over the mainline llama.cpp project, particularly in hybrid multi-GPU setups. Benchmarks on the user's system, which includes an Intel Core i9-10920X and four NVIDIA GPUs, show mainline significantly outperforming ik_llama.cpp in both prompt evaluation and token generation speeds. The user suspects ik_llama.cpp might be optimized for CPU-only or single-GPU inference, and its custom threadpool may not scale well with pipelined layer splits across heterogeneous GPUs, unlike mainline's CUDA graph caching. AI

IMPACT Performance differences in local LLM inference frameworks can impact user experience and hardware optimization choices.

RANK_REASON User-generated technical discussion and benchmark comparison of two software forks.

Read on r/LocalLLaMA →

• ik_llama.cpp
• Intel Core i9-10920X
• llama.cpp
• main line
• NVIDIA P102-100
• Qwen 3.8 Flash-Next 177B Uncensored
• RTX 3060 12GB

• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →