arXiv:2610.11226v1 Announce Type: new Abstract: Weight-only post-training quantization (PTQ) relies heavily on reconstruction loss minimization to preserve model quality at low precision. We show that the weights favored by minimizing this loss need not yield better model perform…
arXiv cs.LG
TIER_1English(EN)·Guanhua Ding, Zi Wang, Ruichao Li, Jack Liu·
arXiv:2610.09212v1 Announce Type: new Abstract: The best two-bit weight quantizers for large language models, such as QTIP and Proteus, rotate each weight matrix by a random orthogonal transform, which must be undone at every decoding step, then encode it with a trellis or lattic…
arXiv:2610.07457v1 Announce Type: cross Abstract: Fine-grained mixed-precision quantization promises efficient large language model inference, but local precision choices can conflict with regular GPU storage and computation units. This precision-boundary mismatch limits the tran…
Serving a large language model (LLM) across a fleet of deployments requires several weight-precision operating points. Multi-precision formats serve them all from one stream whose prefixes are valid lower-precision codes, instead of storing multiple copies. We present StagQ, a mu…
<p>I've been reading about LLM quantization for a while, and almost every explanation I found stopped at the same sentence: "it makes the model smaller."</p> <p>Fine. But <em>how</em>? And what do you give up?</p> <p>So I wrote my notes down, got stuck on a few things, and built …