autoregressive decoding
PulseAugur coverage of autoregressive decoding — every cluster mentioning autoregressive decoding across labs, papers, and developer communities, ranked by signal.
1 day(s) with sentiment data
-
Inception Labs' Mercury 3 diffusion LLM hits 1,000 tokens/sec
Inception Labs has released Mercury 3, a diffusion-based large language model that achieves significantly higher throughput than traditional autoregressive models. This new model can generate approximately 1,068 tokens …
-
DScout paper boosts LLM inference speed with optimized speculative decoding
A new paper introduces DScout, a method to improve the speed of speculative decoding in large language models. DScout optimizes both the training and inference phases of diffusion drafters, which are used to propose tok…
-
New S2-MoE framework boosts MoE model efficiency on edge devices
Researchers have developed S2-MoE, a novel framework designed to make Mixture-of-Experts (MoE) models more efficient for inference on edge devices. This approach addresses the challenges of memory and bandwidth constrai…
-
Bole system accelerates hybrid-attention LLM inference with tree speculation
Researchers have developed Bole, a new system designed to accelerate inference for hybrid-attention large language models. These models combine full attention with recurrent linear attention to manage long contexts more…
-
LLM Inference Costs Driven by Memory Bandwidth, Not Compute
The primary cost driver for large language model inference is not computational power (FLOPs) but memory bandwidth, particularly for handling the KV cache during autoregressive decoding. Optimizing for compute utilizati…
-
New CLP method accelerates LLM inference without quality loss
Researchers have developed a new method called Collocation-Length Prediction (CLP) to accelerate large language model inference. CLP addresses a core issue in multi-token prediction (MTP) where the prediction head for s…