speculative decoding
PulseAugur coverage of speculative decoding — every cluster mentioning speculative decoding across labs, papers, and developer communities, ranked by signal.
- used by alphaXiv 90%
- used by large language model 90%
- used by large-language models 90%
- used by DagsHub 70%
- instance of FLASH 70%
- used by CatalyzeX 70%
- used by vLLM 70%
- instance of CatalyzeX 70%
- developed by FLASH 70%
- instance of alphaXiv 70%
- instance of large language model 70%
- developed Multi Token Prediction 70%
5 day(s) with sentiment data
-
Speculative decoding can slow LLMs if acceptance rate is too low
Speculative decoding, a technique intended to speed up large language model inference, can paradoxically slow down performance if not configured correctly. The method involves a smaller "draft" model generating candidat…
-
GrowMTP trains draft heads within RL loop, accelerating LLM training
Researchers have developed GrowMTP, a novel method that trains a draft head for speculative decoding entirely within the reinforcement learning (RL) loop. This approach eliminates the need for pre-training draft heads s…
-
New speculative decoding methods boost LLM inference speed · 7 sources tracked
Researchers are advancing speculative decoding techniques for large language models to improve inference speed. Two new arXiv papers, ECHO and LoopSpec, introduce hierarchical and pipelined approaches, respectively, to …
-
SpecGuard uses speculative decoding for free LLM backdoor detection
Researchers have developed SpecGuard, a novel method for detecting backdoors in large language models during inference. This technique repurposes speculative decoding, a process typically used to speed up model generati…
-
New research tackles LLM efficiency with NVM quantization and on-device power scaling
Two new research papers explore methods to improve the efficiency of large language models (LLMs). The first paper, "Interface-Aware KV Cache Quantization for Dense On-Chip NVM in Long-Context LLM Decoding," proposes a …
-
vLLM adds speculative decoding for AMD GPUs, boosting inference speed
vLLM has implemented speculative decoding for AMD GPUs, a technique that allows for faster inference by having a smaller draft model propose tokens that a larger target model then verifies. This feature, optimized for A…
-
New methods enhance LLM inference speed via speculative decoding
Researchers are developing advanced techniques for speculative decoding to accelerate large language model (LLM) inference. One approach, X-CoSD, focuses on efficient communication between small on-device models and lar…
-
Speculative decoding observed with distilled models, prompting n-gram integration questions
A Reddit user observed speculative decoding in action when running a distilled model, noting that predictable phrases were generated instantly. This observation led to a question about combining speculative decoding wit…
-
New research enhances LLM speculative decoding for speed and accuracy
Multiple research papers are exploring advancements in speculative decoding for large language models (LLMs), aiming to improve inference speed and output quality. One approach, SpecPV, uses partial verification of key-…
-
ik_llama.cpp adds DFlash 2 speculative decoding and new model support
A pull request has been submitted to the ik_llama.cpp project, introducing DFlash 2 speculative decoding. This update also includes support for IQ4_KS and IQ4_KT quantization formats on RDNA3 GPUs via HIP, Vulkan, and i…
-
AsymSpec framework enhances agentic LLM efficiency with asymmetric decoding
Researchers have introduced AsymSpec, a novel framework for asymmetric speculative decoding designed to improve the efficiency of agentic Large Language Models (LLMs). This method addresses the accuracy-inference cost t…
-
ResiSpec framework boosts LLM speculative decoding efficiency
Researchers have introduced ResiSpec, a new framework designed to enhance the efficiency of speculative decoding in Large Language Models (LLMs). Speculative decoding typically uses a smaller model to predict future tok…
-
ResiSpec framework boosts LLM speculative decoding efficiency
Researchers have introduced ResiSpec, a new framework designed to enhance the efficiency of speculative decoding in large language models. Speculative decoding typically uses a draft model to predict future tokens, whic…
-
New research explores parallel drafting for speculative decoding in LLMs
Two new research papers explore advancements in speculative decoding for large language models, focusing on improving efficiency and coherence in parallel drafting. The first paper surveys the applicability of block-par…
-
New methods accelerate LLM inference with speculative decoding · 7 sources tracked
Researchers are developing new methods to accelerate the inference speed of large language models (LLMs) through speculative decoding. DARTree and SPADE are two such approaches, with DARTree focusing on tree-based specu…
-
Speculative Decoding Matures, Accelerating LLM Inference
Speculative decoding, a technique for accelerating LLM inference, has matured significantly, with frameworks adopting it and users reporting impressive performance gains. While the core concept has existed for years, it…
-
Apple unveils ARBITRAGE to boost LLM reasoning efficiency
Apple Machine Learning Research has introduced ARBITRAGE, a new framework designed to enhance the efficiency of large language models (LLMs) during reasoning tasks. Traditional speculative decoding methods often struggl…
-
Speculative Decoding Explained: GPU Hardware Limits Accelerate LLM Inference
This article explains speculative decoding, a technique that speeds up Large Language Model (LLM) inference by leveraging the physical limitations of graphics processing units (GPUs). The core idea is that while increas…
-
Speculative decoding boosts on-device LLM speed by reducing memory reads
Speculative decoding is a technique designed to accelerate the text generation speed of large language models on resource-constrained devices. This method involves a smaller, faster draft model proposing multiple tokens…
-
New research explores LLM efficiency and reasoning improvements
Several research papers explore methods to enhance the efficiency and reliability of large language models (LLMs). Hugging Face's LFM2.5-DSpark demonstrates up to 3.2x faster inference speeds by using speculative decodi…