Researchers are developing new methods to accelerate large language model (LLM) inference through speculative decoding. ChainFlow, for instance, uses flow-matching on latent states to improve drafting speed by up to 39% over existing methods. ValuSpec offers a plug-and-play mechanism to filter low-quality candidate branches before verification, achieving up to 2.38x speedup. Other approaches like HyFlash and D3D focus on optimizing drafter capacity and decoupling dependencies for more efficient parallel processing, with D3D demonstrating up to 7.48x speedup on specific benchmarks.
AI
IMPACT
These advancements in speculative decoding promise significant speedups for LLM inference, potentially lowering costs and enabling broader real-time applications.
RANK_REASON
Multiple research papers introducing novel methods for speculative decoding in LLMs.
arXiv:2611.08669v1 Announce Type: cross Abstract: Speculative decoding has become a standard way to accelerate LLM inference: a lightweight drafter proposes tokens that the target verifies in one parallel pass. Existing drafters face trade-offs that limits their speedups: autoreg…
arXiv:2611.07263v1 Announce Type: new Abstract: Parallel block drafting accelerates speculative decoding by predicting multiple future positions in a single backbone pass, but on-device deployment constrains both drafter capacity and context storage. Under tight memory budgets, d…
arXiv:2610.31338v2 Announce Type: replace Abstract: Tree-based speculative decoding accelerates large language model inference by verifying multiple candidate branches in parallel. However, in practical serving scenarios, similar requests repeatedly induce overlapping candidate b…
arXiv:2611.07845v1 Announce Type: cross Abstract: Masked diffusion models generate text by reversing a stochastic masking process. Unlike autoregressive LLMs, which generate text one token at a time, they can reveal many tokens in parallel, at the price of an error, since tokens …
arXiv:2611.05611v1 Announce Type: new Abstract: Speculative decoding (SD) speeds up large language model (LLM) serving, but its gain depends on the workload's \emph{shape}, such as batch size and context length, and on its \emph{content}. A model-based drafter proposes a short bl…
arXiv:2611.05894v1 Announce Type: new Abstract: Recycling a target model's own predictions offers a training-free source of proposals for speculative decoding. In vision-language models, however, existing recycling indexes predictions by the last token alone, conflating predictio…
arXiv:2611.06425v1 Announce Type: new Abstract: Target-conditioned diffusion drafters predict candidate tokens in parallel, yet each new block must wait for updated target-model hidden states. This forces drafting and verification to run sequentially, limiting speedup even though…
arXiv cs.LG
TIER_1English(EN)·Hoang Anh Duy Le, Huibing Dong, Alireza Khadem, Ziteng Sun, Xiang Song, Yizheng Huang, Soroush Radpour, Anshumali Shrivastava, Ye He·
arXiv:2611.05724v1 Announce Type: new Abstract: Speculative decoding accelerates large language model inference with a small drafter, whose per-guess acceptance rate determines the achievable speedup. Feature-level drafters share a design choice: every token, easy or hard, is dra…
Parallel block drafting accelerates speculative decoding by predicting multiple future positions in a single backbone pass, but on-device deployment constrains both drafter capacity and context storage. Under tight memory budgets, drafters must model long-range dependencies and f…
D3D is a standalone diffusion drafter that removes target-model hidden-state dependencies to enable asynchronous speculative decoding with high speedup.
arXiv:2610.41177v1 Announce Type: new Abstract: LLM agents execute tasks through repeated model calls over evolving histories of reasoning, actions, and tool feedback. Aggregate speculative-decoding metrics merge these distinct decisions, providing limited guidance on where draft…
SpecOPD accelerates speculative decoding by parallelizing on-policy draft supervision across independent branches, improving acceptance length and training throughput across diverse model architectures and tasks.
SpecPilot autonomously discovers deployment-specific algorithms and system optimizations to maximize speculative decoding speedups across diverse models, drafters, and serving regimes without quality loss.
arXiv cs.AI
TIER_1English(EN)·Abhishek Malvankar, Paul Castro, Nick Mitchell, Lionel Villard·
arXiv:2610.38968v1 Announce Type: cross Abstract: Speculative decoding speeds up large language model (LLM) serving only while the GPU has spare compute, so the best draft length, the number of tokens a small drafter guesses per step, shrinks as the batch grows. vLLM lets operato…
arXiv:2610.38981v1 Announce Type: cross Abstract: Speculative decoding (SD) accelerates autoregressive inference by leveraging the memory bound nature of decoding, verifying multiple draft tokens in a single forward pass. For mixture-of-experts (MoE) models, prior work predicts a…
arXiv cs.AI
TIER_1English(EN)·Bole Ma, Ayesha Afzal, Jan Eitzinger, Harald Koestler, Gerhard Wellein·
arXiv:2610.40287v1 Announce Type: cross Abstract: Large language model serving is limited by facility power. Energy per token determines how many tokens a datacenter can serve, while a latency target bounds how quickly each user receives them. During decode, every weight is read …
arXiv cs.CV
TIER_1English(EN)·Hong Nie, Zhangchi Gao, Yuqian Liang, Jun Yu, Bo Wang, Jianqing Sun, Shengping Liu·
arXiv:2611.01617v2 Announce Type: replace-cross Abstract: Speculative decoding accelerates vision-language model (VLM) inference without altering the output distribution, yet two factors limit its gains. Existing drafters either lack the image evidence required for visually groun…
<h2> TL;DR </h2> <p>Tencent가 11월 7일 HF에 올린 <code>Qwen3.8-27B-DSpine-b16</code>은 Qwen3.8-27B의 속도를 높이기 위한 <strong>투기 디코딩 초안 모델</strong>입니다. 초안 모델이 한 번에 16토큰을 병렬로 제안하고, 큰 모델이 그것을 한꺼번에 검증하는 방식입니다. 카드에는 비교 대상인 DFlash2보다 수락 길이가 길다는 표와, 자기회귀 디코딩 대비 최대 3.84배라는 처리량 표가 있습니다.</p> <p>저희는 이 체…
