llama.cpp patch boosts Qwen context length on AMD GPUs to 149K

A user on Reddit has developed a patch for llama.cpp that significantly increases the context length for models running on AMD GPUs. By optimizing the Multi Token Prediction (MTP) buffer allocation, the patch allows for context lengths of up to 149K tokens for the Qwen 27B model, a substantial increase from the default 64K. This optimization is particularly beneficial for ROCm configurations with multiple GPUs, offering improved performance and context length compared to Vulkan backends, though Vulkan still offers some VRAM savings. AI

IMPACT Enables longer context windows for local LLM deployments on AMD hardware, potentially improving performance on complex tasks.

RANK_REASON User-developed patch for open-source software improving performance.

Read on r/LocalLLaMA →

• AMD
• llama.cpp
• Multi Token Prediction
• Qwen

• infra
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →