llama.cpp adds CUDA FWHT for faster KV cache quantization

A pull request to the llama.cpp project introduces a CUDA implementation of the Fast Walsh-Hadamard Transform (FWHT). This optimization, developed by user am17an, aims to speed up operations when quantizing the key-value cache. Initial benchmarks show modest performance gains, with a 1-2% boost in processing power (pp) and a 7-9% increase in token generation (tg) for the Gemma 4 26B model. AI

影响 Improves inference efficiency for local LLM deployments by optimizing KV cache operations.

排序理由 This is a pull request for a specific optimization within an open-source project, not a major model release or industry-shaping event.

在 r/LocalLLaMA 阅读 →

• am17an
• CUDA
• Fast Walsh-Hadamard Transform
• Gemma 4 26B
• llama.cpp

• 基础设施
• 其他

AI 生成摘要 · Google Gemini · 来自 1 个来源。 我们如何撰写摘要 →