A Reddit user has proposed a novel speculative decoding technique to run large language models that exceed GPU VRAM. The method involves keeping a 2-bit version of the model on the GPU for drafting tokens and storing the remaining weights in system RAM. A CPU kernel then verifies these drafted tokens, with only activations transferred over PCIe. This approach reportedly offers quality comparable to 8-bit quantization while significantly reducing VRAM requirements and improving draft acceptance rates compared to existing methods. AI
IMPACT Enables running larger language models on consumer hardware by optimizing VRAM usage.
RANK_REASON User-submitted research on a novel technique for running LLMs. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
