研究人员正在开发先进的量化技术，以提高大型语言模型 (LLM) 的效率。AutoRound、LATMiX 和 GSQ 等新方法旨在减小模型大小和计算需求，从而能够在功能较弱的硬件上进行部署。这些方法侧重于优化模型权重和激活在较低比特宽度下的表示方式，其中一些方法已达到与更高精度模型相当的准确性。创新包括用于训练后量化的新颖校准策略和用于提高鲁棒性的可学习仿射变换。 AI
影响 能够更有效地在资源受限的设备上部署 LLM，从而可能降低推理成本并提高可访问性。
排序理由 多篇 arXiv 论文和 Hugging Face 博客文章详细介绍了 LLM 量化方面的新研究和工具。
AI 生成摘要 · Google Gemini · 来自 16 个来源。 我们如何撰写摘要 →
Large language models (LLMs) are costly to deploy due to their large memory footprint and high inference cost. Weight-activation quantization can reduce these costs, but low-bit activation quantization remains difficult because activation outliers induce large quantization error.…
arXiv:2605.01164v1 Announce Type: new Abstract: This position paper argues that LLMs should not yet be credited with decision explanation. This matters because recent work increasingly treats accurate behavioral prediction, plausible rationales, and outcome-conditioned reasoning …
arXiv:2605.00662v1 Announce Type: cross Abstract: Sequence learning reduces to similarity-based retrieval over a temporally indexed representation space, a constraint on any sequence model, not a property of a specific architecture. We show that a spiking Sparse Distributed Memor…
Sequence learning reduces to similarity-based retrieval over a temporally indexed representation space, a constraint on any sequence model, not a property of a specific architecture. We show that a spiking Sparse Distributed Memory sequence machine (2007) and the transformer (201…
arXiv:2604.24008v1 Announce Type: new Abstract: Post-Training Quantization (PTQ) compresses large language models to low bit-widths using a small calibration set, and its quality depends strongly on which samples are chosen. We identify a failure mode in which calibration samples…
arXiv:2602.17681v2 Announce Type: replace-cross Abstract: Post-training quantization (PTQ) is a widely used approach for reducing the memory and compute costs of large language models (LLMs). Recent studies have shown that applying invertible transformations to activations can si…
arXiv:2410.21548v3 Announce Type: replace Abstract: Large language models have drastically changed the prospects of AI by introducing technologies for more complex natural language processing. However, current methodologies to train such LLMs require extensive resources including…
Weight quantization has become a standard tool for efficient LLM deployment, especially for local inference, where models are now routinely served at 2-3 bits per parameter. The state of the art is currently split into two sets of methods: simple scalar quantization techniques, s…
arXiv:2410.24214v3 Announce Type: replace-cross Abstract: Mixed precision quantization has become an important technique for optimizing the execution of deep neural networks (DNNs). Certified robustness, which provides provable guarantees about a model's ability to withstand diff…
arXiv:2408.00923v2 Announce Type: replace Abstract: This paper explores a novel paradigm in low-bit (i.e. 4-bits or lower) quantization, differing from existing state-of-the-art methods, by framing optimal quantization as an architecture search problem within convolutional neural…
TurboQuant: A First-Principles Walkthrough A brisk, brilliantly coded tutorial on vector quantisation: how far you can push compression on model KV caches and embeddings without breaking what matters. The interactive slider(...) # ai # javascript # ml # quantization # tutorial # …