PulseAugur
EN
LIVE 00:42:24

LLM Quantization Failure: Outliers Zero Out 99.9% of Data

A technical blog post details a critical failure mode in INT4 and INT8 quantization for large language models (LLMs) caused by outliers. When a single outlier value is present in a block of 4,096 floating-point numbers, it can cause 4,095 of those numbers to be quantized to zero in INT4, effectively destroying the signal from the non-outlier values. This phenomenon explains why per-tensor INT4 quantization is rarely used, why group sizes are limited in methods like GPTQ, and the purpose of parameters like `llm_int8_threshold`. The post suggests that fixes involve preventing outliers from dominating the scale calculation, such as using group-wise scales, mixed-precision decomposition, or methods like SmoothQuant that adjust outlier magnitudes. AI

IMPACT Highlights a critical vulnerability in LLM quantization that can lead to significant performance degradation or incorrect outputs, necessitating careful parameter tuning or alternative methods.

RANK_REASON Technical deep-dive into a specific failure mode of LLM quantization techniques. [lever_c_demoted from research: ic=1 ai=1.0]

Read on dev.to — LLM tag →

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →

LLM Quantization Failure: Outliers Zero Out 99.9% of Data

How we ranked this

Signal score
17 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Tool
Technical deep-dive into a specific failure mode of LLM quantization techniques. [lever_c_demoted from research: ic=1 ai=1.0]
Source corroboration
Single-source cluster
Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics
infra
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
Breaking (< 6h)
Fresh story with cross-source coverage still developing. Ranking may shift as more sources report.

Full methodology in our editorial standards.

COVERAGE [1]

  1. dev.to — LLM tag TIER_1 English(EN) · jidonglab ·

    INT4 Absmax Quantization: One Outlier Zeroed 4,095 of 4,096 Values

    <p>I planted one number in a vector of 4,096 floats. I set it to 60, quantized the vector to INT4 with absmax scaling, and looked at what was left.</p> <p>All 4,095 of the other values were exactly zero.</p> <p>Not "a bit noisy." Zero. The whole row went down to one integer level…