speculative decoding
PulseAugur coverage of speculative decoding — every cluster mentioning speculative decoding across labs, papers, and developer communities, ranked by signal.
- instance of Gotit.pub 90%
- uses large-language models 90%
- instance of DSpark 90%
- used by large language model 90%
- used by large-language models 90%
- used by DagsHub 70%
- used by alphaXiv 70%
- used by CatalyzeX 70%
- instance of alphaXiv 70%
- instance of CatalyzeX 70%
- used by vLLM 70%
- used by ScienceCast 70%
13 day(s) with sentiment data
-
Split-KV method boosts diffusion drafter training efficiency
Researchers have developed Split-KV, a novel context parallelism method designed to enhance the training efficiency of diffusion drafters. This technique addresses the challenge of load balancing in diffusion drafters, …
-
New TVP loss function enhances speculative decoding for faster LLM inference
Researchers have developed TVP, a novel loss function family designed to improve speculative decoding in large language models. This method aims to accelerate inference by optimizing the acceptance rate of proposed toke…
-
Nvidia RTX Spark laptops face Linux performance issues due to encrypted firmware
Nvidia's RTX Spark laptops, designed for AI development, experience a significant performance drop when running Linux due to encrypted firmware that limits GPU clock speeds. While Windows allows the GPU to reach over 2 …
-
New speculative decoding methods boost LLM inference speed · 9 sources tracked
Researchers are developing new methods to accelerate large language model (LLM) inference through speculative decoding. ChainFlow, for instance, uses flow-matching on latent states to improve drafting speed by up to 39%…
-
New speculative decoding method optimizes Mixture-of-Experts models
Researchers have developed a new method for speculative decoding in Mixture-of-Experts (MoE) models, focusing on optimizing expert byte requests and kernel bandwidth. Their approach aims to reduce the computational cost…
-
XPress enhances speculative decoding for diffusion models
Researchers have developed xPress, a novel method to improve speculative decoding in diffusion drafters. This technique addresses the issue of independently sampled tokens in diffusion models, which can lead to jointly …
-
MoE-SpAc framework enhances MoE model inference efficiency on edge devices
Researchers have developed MoE-SpAc, a novel framework designed to improve the efficiency of Mixture-of-Experts (MoE) model inference on heterogeneous edge devices. This system repurposes speculative decoding techniques…
-
NgramRL accelerates LLM training with rollout-aware n-gram speculative decoding
Researchers have developed NgramRL, a novel method to accelerate reinforcement learning (RL) post-training for large language models. This technique addresses the significant cost of autoregressive rollout generation, w…
-
New SPOT method accelerates diffusion language models by predicting token generation order
Researchers have developed a new method called SPOT (Speculating Positions Order Trajectory) to significantly accelerate diffusion language models (DLMs). SPOT addresses the speed-quality trade-off in DLMs by predicting…
-
LLM inference optimizations perform differently on Apple Silicon vs. NVIDIA GPUs
A new research paper evaluates the effectiveness of common LLM inference optimizations when applied to Apple Silicon, finding that techniques designed for server-class NVIDIA GPUs do not always translate effectively. Wh…
-
RIFT improves KV cache compression for speculative decoding in LLMs
Researchers have introduced RIFT, a novel KV cache compression strategy designed to improve the efficiency of speculative decoding in large language models. RIFT addresses the issue of "decode staleness" where compressi…
-
New research improves speculative decoding efficiency for language models
A new research paper explores how to improve speculative decoding, a technique used in language models to speed up text generation. The study, published on arXiv, investigates whether the probability distributions gener…
-
Draft-training objectives show diminishing returns in speculative decoding
A new research paper from arXiv explores the effectiveness of draft-training objectives in speculative decoding for large language models. The study found that while these objectives initially show gains over standard c…
-
Speculative decoding gains depend on acceptance cost, study finds
Researchers have conducted a controlled study on speculative decoding, a technique that can improve the throughput of language models. The study found that the benefits of speculative decoding are highly dependent on th…
-
Specter system accelerates MoE inference with quantization and speculative offloading
Researchers have developed Specter, a novel inference system designed to accelerate Mixture of Experts (MoE) models, which often exceed the memory capacity of a single accelerator. Specter combines quantization with spe…
-
Speculative decoding speeds up LLMs by changing GPU workload
Speculative decoding in large language models is fast not because it reduces computational work, but because it shifts the type of work performed by GPUs. While the total number of floating-point operations may increase…
-
New SPOT method accelerates diffusion language models by predicting token order
Researchers have developed a new method called Speculating Positions Order Trajectory (SPOT) to enhance the speed of diffusion language models (DLMs) without sacrificing quality. DLMs, while promising parallel text gene…
-
Speculative decoding faces length inflation, study finds
A new research paper published on arXiv titled "Acceptance Is Not Speed: Length Inflation in Training-Free Lossy Speculative Decoding" identifies a phenomenon called length inflation in speculative decoding. This occurs…
-
Research: Short-context drafters can handle 2M tokens without retraining
A new research paper proposes that long-context drafting in language models does not necessitate specialized long-context drafters. The authors argue that by freezing existing short-context drafters and employing infere…
-
DLite framework improves agentic AI inference speed by eliminating historical context
Researchers have introduced DLite, a novel history-free speculative decoding framework designed to enhance the efficiency of agentic AI workloads. Unlike existing methods that maintain historical context, DLite reuses h…