arXiv:2610.21436v1 Announce Type: new Abstract: Device-cloud speculative decoding reduces server computation, but growing a deep draft tree with a small language model (SLM) requires repeated passes on a mobile neural processing unit (NPU). We present HyCo, which assigns depth to…
arXiv:2610.20175v1 Announce Type: new Abstract: Parallel drafters such as DFlash and DSpark reduce proposal latency, but early mismatches limit how much of each block can be accepted. Retaining alternatives introduces parent states and candidate choices that reference-conditioned…
arXiv:2610.20392v1 Announce Type: new Abstract: Speculative decoding accelerates LLM generation through lightweight drafting and parallel verification. It is particularly promising for long inputs, where KV-cache access costs can be amortized over multiple accepted tokens. Yet mo…
arXiv:2610.13708v1 Announce Type: new Abstract: Long-context serving runs out of KV memory before the hardware reaches its useful throughput. HiSparse stores history in host memory and fetches only the selected entries, but it uses one cache budget at every load and does not hand…
arXiv:2610.12939v1 Announce Type: cross Abstract: Decoder hidden states from different layers of large language models (LLMs) often encode complementary information. However, existing speculative decoding methods typically rely on a small set of target-model layers chosen using f…
arXiv:2610.13202v1 Announce Type: cross Abstract: Fast rollout generation for long-context reinforcement learning with verifiable rewards (RLVR) requires reducing KV-cache traffic without changing the response groups used for policy optimization. In our profiled RLVR workloads, a…
arXiv cs.CL
TIER_1English(EN)·Roy Eisenstadt, Harel Lidar, Tal Gonen, Omer Luxembourg, Itamar Zimerman, Lior Wolf, Eli David, Ilan Naiman·
arXiv:2610.12570v1 Announce Type: new Abstract: Speculative decoding accelerates large language model inference by using a lightweight drafter to propose multiple tokens that are subsequently verified in parallel by the target model. Recent diffusion-based drafters make this para…
arXiv cs.AI
TIER_1English(EN)·Shijing Hu, Xuancheng Ren, Zhihui Lu, Pan Zhou·
arXiv:2610.11750v1 Announce Type: new Abstract: Tree-based speculative decoding verifies multiple draft continuations in one target-model pass, but finite trees built from draft scores face a fundamental draft-target mismatch. We ask whether better exact verification can increase…
arXiv cs.AI
TIER_1English(EN)·Adam Y. J. Jones, Yu Yuan, Sergio Maffeis·
arXiv:2610.10929v1 Announce Type: cross Abstract: Speculative decoding is a popular technique for increasing the speed and reducing the costs of large language model (LLM) inference by verifying multiple draft tokens in a single target-model forward pass. The resulting benefit de…
Decoder hidden states from different layers of large language models (LLMs) often encode complementary information. However, existing speculative decoding methods typically rely on a small set of target-model layers chosen using fixed, position-based heuristics. Selecting an opti…
arXiv:2610.10411v1 Announce Type: cross Abstract: Speculative decoding accelerates large language model inference by using a low-cost draft model to propose tokens that the full-size target model verifies in parallel. Parallel and semi-autoregressive (semi- AR) drafters improve d…
Speculative decoding accelerates large language model inference by using a lightweight drafter to propose multiple tokens that are subsequently verified in parallel by the target model. Recent diffusion-based drafters make this paradigm particularly attractive: deep attention-bas…
arXiv:2610.04875v2 Announce Type: replace Abstract: Diffusion large language models (DLLMs) generate text through iterative block denoising, and multi-branch speculative decoding accelerates this process by verifying a main branch together with multiple draft branches in a single…
arXiv cs.AI
TIER_1English(EN)·Yichi Zhang, Zhiqi Wang, Neil Gong, Yuchen Yang·
arXiv:2610.08678v1 Announce Type: cross Abstract: Speculative decoding accelerates inference for a large language model (LLM), referred to as the \emph{target model}, by first using a smaller model, referred to as the \emph{draft model}, to generate candidate tokens and then veri…
arXiv:2610.07659v1 Announce Type: new Abstract: Speculative decoding accelerates autoregressive generation in large language models. In each drafting stage, a lightweight draft model proposes tokens that the target model subsequently verifies. With increasingly capable draft mode…
arXiv:2610.01108v2 Announce Type: replace Abstract: Retrieval-based speculative decoding (SD) drafts tokens by copying continuations from existing text, which suits coding agents that repeatedly reproduce code, logs, and earlier attempts. Yet existing methods fall short in agent …
arXiv:2610.07822v1 Announce Type: new Abstract: Speculative decoding accelerates autoregressive generation by using a lightweight draft model to propose multiple tokens that are verified by a target model in parallel. However, the standard acceptance rule focuses on exact distrib…
Diffusion large language models (DLLMs) generate text through iterative block denoising, and multi-branch speculative decoding accelerates this process by verifying a main branch together with multiple draft branches in a single forward pass. While prior DLLM acceleration methods…
Speculative decoding accelerates autoregressive generation in large language models. In each drafting stage, a lightweight draft model proposes tokens that the target model subsequently verifies. With increasingly capable draft models, we find that the target model frequently acc…
<p>I added a draft model to my vLLM server, sent one curl request, and watched the tokens fly. Then I ran the load test with 64 concurrent users and total throughput went <em>down</em>. Same GPU, same target model, same prompts. The only change was one flag that every blog post s…