autoregressive decoding
PulseAugur coverage of autoregressive decoding — every cluster mentioning autoregressive decoding across labs, papers, and developer communities, ranked by signal.
-
New S2-MoE framework boosts MoE model efficiency on edge devices
Researchers have developed S2-MoE, a novel framework designed to make Mixture-of-Experts (MoE) models more efficient for inference on edge devices. This approach addresses the challenges of memory and bandwidth constrai…
-
Bole system accelerates hybrid-attention LLM inference with tree speculation
Researchers have developed Bole, a new system designed to accelerate inference for hybrid-attention large language models. These models combine full attention with recurrent linear attention to manage long contexts more…
-
LLM Inference Costs Driven by Memory Bandwidth, Not Compute
The primary cost driver for large language model inference is not computational power (FLOPs) but memory bandwidth, particularly for handling the KV cache during autoregressive decoding. Optimizing for compute utilizati…
-
New CLP method accelerates LLM inference without quality loss
Researchers have developed a new method called Collocation-Length Prediction (CLP) to accelerate large language model inference. CLP addresses a core issue in multi-token prediction (MTP) where the prediction head for s…