CUDA: add fast walsh-hadamard transform by am17an · Pull Request #23615 · ggml-org/llama.cpp
A pull request to the llama.cpp project introduces a CUDA implementation of the Fast Walsh-Hadamard Transform (FWHT). This optimization, developed by user am17an, aims to speed up operations when quantizing the key-value cache. Initial benchmarks show modest performance gains, with a 1-2% boost in processing power (pp) and a 7-9% increase in token generation (tg) for the Gemma 4 26B model. AI
IMPACT Improves inference efficiency for local LLM deployments by optimizing KV cache operations.