A technical blog post details a critical failure mode in INT4 and INT8 quantization for large language models (LLMs) caused by outliers. When a single outlier value is present in a block of 4,096 floating-point numbers, it can cause 4,095 of those numbers to be quantized to zero in INT4, effectively destroying the signal from the non-outlier values. This phenomenon explains why per-tensor INT4 quantization is rarely used, why group sizes are limited in methods like GPTQ, and the purpose of parameters like `llm_int8_threshold`. The post suggests that fixes involve preventing outliers from dominating the scale calculation, such as using group-wise scales, mixed-precision decomposition, or methods like SmoothQuant that adjust outlier magnitudes. AI
IMPACT Highlights a critical vulnerability in LLM quantization that can lead to significant performance degradation or incorrect outputs, necessitating careful parameter tuning or alternative methods.
RANK_REASON Technical deep-dive into a specific failure mode of LLM quantization techniques. [lever_c_demoted from research: ic=1 ai=1.0]
- AbsMax
- Activation Aware Quantization
- bitsandbytes
- GPTQ
- Int4
- Int8
- LLM
- llm_int8_threshold
- NumPy
- SmoothQuant
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →