Q8_0 isn't slow because of swap
A benchmark of Llama 3.1 8B on an Apple M4 Mac Mini with 16GB unified memory revealed that the Q8_0 quantization, despite fitting entirely in memory, suffers from slow token generation due to memory bandwidth limitations. The analysis showed that the 8-bit weights saturate the memory bus, causing the GPU to spend most of its time transferring data rather than computing. The study identified Q4_K_M as a practical sweet spot, offering nearly the same quality as Q8_0 but at a significantly faster speed without hitting swap. AI
IMPACT Identifies memory bandwidth as a key bottleneck for local LLM deployment, influencing hardware choices and quantization strategies for enterprise applications.