New research enhances LLM inference speed with advanced speculative decoding techniques · 8 sources tracked
ByPulseAugur Editorial·[12 sources]·
Researchers are exploring advanced techniques to accelerate large language model (LLM) inference through speculative decoding. New methods like "Functional Reconstruction" aim to improve the agreement between draft and target models by optimizing attention functions, while "SparseSpec-L" utilizes a dynamically sparsified KV cache and per-head attention statistics for efficiency. Other work revisits "lossy verification" to analyze trade-offs and failure modes, classifying approaches into truncation-based and collaborative verification. Additionally, "AngelSpec" offers a unified training framework for different speculative decoding structures, and "DraftExpert" focuses on optimizing inference for Mixture-of-Experts (MoE) models on end-devices by training lightweight draft experts.
AI
IMPACT
These advancements in speculative decoding aim to significantly reduce LLM inference latency and computational costs, potentially enabling more efficient deployment on edge devices and faster real-time applications.
RANK_REASON
Multiple academic papers published on arXiv detailing new research in speculative decoding for LLMs.
arXiv:2607.27269v1 Announce Type: new Abstract: Multi-head latent attention (MLA) is increasingly important for long-context LLM inference because compact latent states replace the growing key-value (KV) cache and reduce decoding memory traffic. Yet most capable open checkpoints …
arXiv:2607.27735v1 Announce Type: new Abstract: Speculative decoding alleviates the memory-bandwidth bottleneck in large language model inference, but its acceleration is jointly constrained by drafting overhead, token acceptance, and speculation length. We present a unified effi…
arXiv:2607.26627v1 Announce Type: new Abstract: Speculative Decoding (SD) accelerates large language model inference by allowing a lightweight draft model to propose tokens that are subsequently verified in parallel by a larger target model. Recent approaches introduce lossy veri…
Speculative Decoding (SD) accelerates large language model inference by allowing a lightweight draft model to propose tokens that are subsequently verified in parallel by a larger target model. Recent approaches introduce lossy verification schemes to further improve efficiency b…
arXiv:2607.25852v1 Announce Type: new Abstract: Speculative decoding accelerates large language model inference without changing the target distribution, but no single drafting structure performs best across real-world workloads. Autoregressive multi-token prediction (MTP) is a l…
Speculative Decoding (SD) accelerates large language model inference by allowing a lightweight draft model to propose tokens that are subsequently verified in parallel by a larger target model. Recent approaches introduce lossy verification schemes to further improve efficiency b…
arXiv:2607.24434v1 Announce Type: cross Abstract: Large Mixture-of-Experts (MoE) language models are attractive for end-device deployment because only a small subset of experts is active per token, but their routed expert weights often exceed accelerator memory. We target latency…
arXiv:2607.22634v1 Announce Type: new Abstract: Diffusion Large Language Models (dLLMs) have emerged as a promising alternative to autoregressive (AR) LLMs, generating tokens in parallel. This makes them effective draft models for speculative decoding (SD), producing an entire bl…
Large Mixture-of-Experts (MoE) language models are attractive for end-device deployment because only a small subset of experts is active per token, but their routed expert weights often exceed accelerator memory. We target latency-critical single-user settings where routed expert…
Speculative decoding, in which a lightweight draft model first generates a draft sequence that is then verified in parallel by the target model, has become a prevalent paradigm for accelerating large language model inference. Recent work such as DFlash further boosts drafting eff…
<p>Tencent has released AngelSpec, an open-source torch-native framework for training speculative-decoding draft models across six architectures. It introduces DFly, a block-diffusion drafter with hybrid target conditioning and a hidden-correction autoregressive head, and integra…
Tencent has open-sourced AngelSpec, a unified training framework for speculative-decoding draft models. The system addresses workload heterogeneity by providing two complementary drafters: an autoregressive MTP model for conversational data and a block-parallel model for code and…
