UltraSketchLLM: Sub-1-Bit LLM Compression via Sketch and Hardware-Friendly Operators
Researchers have developed UltraSketchLLM, a novel method for compressing large language models (LLMs) to sub-1-bit per weight. This technique utilizes data sketching to significantly reduce GPU memory requirements, achieving a compression rate of 0.5 bits per weight. The approach also incorporates hardware-friendly operators, resulting in a 14.9x speedup compared to standard sketching methods while maintaining tolerable performance degradation and low latency. AI
IMPACT Enables deployment of large language models on resource-constrained hardware, potentially broadening access and application.