Researchers are advancing speculative decoding techniques for large language models to improve inference speed. Two new arXiv papers, ECHO and LoopSpec, introduce hierarchical and pipelined approaches, respectively, to optimize draft candidate generation and verification. ECHO uses early-layer bonus logits for rapid draft exploration, while LoopSpec leverages intermediate states in Looped Transformers for pipelined draft generation. Another paper on Orthrus questions the 'lossless' claim of speculative decoding, highlighting the impact of numerical precision on output consistency, and a separate study on Carryover Drafting proposes recycling rejected states to enhance acceptance length and overall speedup.
AI
IMPACT
These advancements in speculative decoding could significantly reduce LLM inference latency and computational costs, enabling wider deployment and more efficient use of large models.
RANK_REASON
Multiple academic papers published on arXiv detailing new methods and analyses of speculative decoding techniques for LLMs.
arXiv:2609.20186v1 Announce Type: new Abstract: Speculative Decoding (SD) has significantly accelerated Large Language Model (LLM) inference, yet existing approaches face a fundamental tradeoff between two drafting strategies: neural drafting and context-based copying. Neural dra…
arXiv:2609.17943v1 Announce Type: cross Abstract: Long-context LLM inference is bottlenecked by attention, whose repeated KV-cache reads make decoding memory-bound. Self-speculative decoding alleviates this by drafting tokens with sparse attention and verifying them with full att…
arXiv:2609.17184v1 Announce Type: cross Abstract: Looped Transformers achieve strong performance with compact parameter sizes by repeatedly applying a shared stack of Transformer blocks across recurrent depths. However, they incur higher decoding latency than standard Transformer…
arXiv:2609.17241v1 Announce Type: new Abstract: While draft-model-free speculative decoding offers a promising path to efficient LLM inference, it is frequently constrained by stale draft candidates and the high computational cost of the verification. To address these challenges,…
While draft-model-free speculative decoding offers a promising path to efficient LLM inference, it is frequently constrained by stale draft candidates and the high computational cost of the verification. To address these challenges, we propose ECHO, a hierarchical dual-loop frame…
arXiv:2609.14717v1 Announce Type: cross Abstract: Speculative decoding accelerates LLM inference by verifying multiple drafted tokens in parallel, allowing a single target forward pass to accept several tokens. By construction, verification computes representations for both accep…
arXiv cs.AI
TIER_1English(EN)·Ilya Koziev, Leonid Sinev, Ivan Oseledets·
arXiv:2609.15504v1 Announce Type: cross Abstract: Orthrus is a hybrid autoregressive-diffusion architecture that accelerates autoregressive language-model inference by generating multiple tokens in parallel while using a frozen autoregressive backbone. Its central claim is that a…
Orthrus is a hybrid autoregressive-diffusion architecture that accelerates autoregressive language-model inference by generating multiple tokens in parallel while using a frozen autoregressive backbone. Its central claim is that an intra-model consensus mechanism enables lossless…
Orthrus achieves lossless speculative decoding only under high numerical precision, with trajectory divergence under BF16 that does not impair downstream benchmarks.
dev.to — LLM tag
TIER_1English(EN)·Manoranjan Rajguru·
<h1> Speculative Decoding in 2026: From EAGLE to DFlash to XPress — The Complete Engineer's Playbook </h1> <p><a class="article-body-image-wrapper" href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-upl…
Speculative Decoding in 2026: From EAGLE to DFlash to XPress — The Complete Engineer's... # ai # llm # machinelearning # programming # software # coding # development # engineering # inclusive # community Speculative Decoding in 2026: From EAGLE to DFlash to XPress — The Complete…
