English(EN)How Lossless Is Lossless Speculative Decoding? The Role of Numerical Precision in Orthrus
新的投机式解码方法提高了LLM推理速度 · 跟踪7个来源
作者PulseAugur 编辑部·[11 个来源]·
研究人员正在推进大型语言模型的投机式解码技术,以提高推理速度。两篇新的arXiv论文,ECHO和LoopSpec,分别引入了分层和流水线方法来优化草稿候选生成和验证。ECHO使用早期层奖励logits进行快速草稿探索,而LoopSpec则利用循环Transformer中的中间状态进行流水线草稿生成。另一篇关于Orthrus的论文质疑了投机式解码的“无损”声明,强调了数值精度对输出一致性的影响,而另一项关于Carryover Drafting的研究提出回收被拒绝的状态以增强接受长度和整体加速。
AI
arXiv:2609.20186v1 Announce Type: new Abstract: Speculative Decoding (SD) has significantly accelerated Large Language Model (LLM) inference, yet existing approaches face a fundamental tradeoff between two drafting strategies: neural drafting and context-based copying. Neural dra…
arXiv:2609.17943v1 Announce Type: cross Abstract: Long-context LLM inference is bottlenecked by attention, whose repeated KV-cache reads make decoding memory-bound. Self-speculative decoding alleviates this by drafting tokens with sparse attention and verifying them with full att…
arXiv:2609.17184v1 Announce Type: cross Abstract: Looped Transformers achieve strong performance with compact parameter sizes by repeatedly applying a shared stack of Transformer blocks across recurrent depths. However, they incur higher decoding latency than standard Transformer…
arXiv:2609.17241v1 Announce Type: new Abstract: While draft-model-free speculative decoding offers a promising path to efficient LLM inference, it is frequently constrained by stale draft candidates and the high computational cost of the verification. To address these challenges,…
While draft-model-free speculative decoding offers a promising path to efficient LLM inference, it is frequently constrained by stale draft candidates and the high computational cost of the verification. To address these challenges, we propose ECHO, a hierarchical dual-loop frame…
arXiv:2609.14717v1 Announce Type: cross Abstract: Speculative decoding accelerates LLM inference by verifying multiple drafted tokens in parallel, allowing a single target forward pass to accept several tokens. By construction, verification computes representations for both accep…
arXiv cs.AI
TIER_1English(EN)·Ilya Koziev, Leonid Sinev, Ivan Oseledets·
arXiv:2609.15504v1 Announce Type: cross Abstract: Orthrus is a hybrid autoregressive-diffusion architecture that accelerates autoregressive language-model inference by generating multiple tokens in parallel while using a frozen autoregressive backbone. Its central claim is that a…
Orthrus is a hybrid autoregressive-diffusion architecture that accelerates autoregressive language-model inference by generating multiple tokens in parallel while using a frozen autoregressive backbone. Its central claim is that an intra-model consensus mechanism enables lossless…
Orthrus achieves lossless speculative decoding only under high numerical precision, with trajectory divergence under BF16 that does not impair downstream benchmarks.
dev.to — LLM tag
TIER_1English(EN)·Manoranjan Rajguru·
<h1> Speculative Decoding in 2026: From EAGLE to DFlash to XPress — The Complete Engineer's Playbook </h1> <p><a class="article-body-image-wrapper" href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-upl…
Speculative Decoding in 2026: From EAGLE to DFlash to XPress — The Complete Engineer's... # ai # llm # machinelearning # programming # software # coding # development # engineering # inclusive # community Speculative Decoding in 2026: From EAGLE to DFlash to XPress — The Complete…