TensorRT-LLM
PulseAugur coverage of TensorRT-LLM — every cluster mentioning TensorRT-LLM across labs, papers, and developer communities, ranked by signal.
4 day(s) with sentiment data
-
Speculative decoding boosts LLM inference speed by up to 50%
A new method for speculative decoding can accelerate Large Language Model (LLM) inference by 20-50% without compromising output quality. This technique involves draft-verification mechanics and is compatible with variou…
-
NVIDIA TensorRT-LLM: Fastest Throughput, But Beware Deployment Costs
NVIDIA's TensorRT-LLM framework offers impressive throughput speeds, but choosing it solely based on headline performance metrics can lead to hidden costs. The article suggests that while TensorRT-LLM is the fastest in …
-
NVIDIA's software stack slashes AI inference token costs on Blackwell platform
NVIDIA is highlighting how its integrated software stack, optimized for its Blackwell platform, significantly reduces the cost per token for AI inference. By coordinating production operations, application acceleration,…
-
New technique speeds up LLM inference by pre-decoding sessions
Researchers have introduced a new technique called speculative pre-positioning to improve the efficiency of stateless inference servers for large language models. This method decodes sessions forward to their next decis…
-
Tokens per Watt to Dictate 2026 GPU and Cooling Decisions
The primary constraint for AI compute in 2026 will shift from raw processing power to efficiency, specifically tokens per watt. This is because inference, which now accounts for the majority of AI compute spend, is fund…
-
LLM Inference Handbook Explains Token Generation and Optimization
This handbook delves into the engineering discipline of Large Language Model (LLM) inference, explaining how models generate tokens and the advanced optimization techniques used in production systems. It covers fundamen…
-
KForge uses LLM agents to auto-generate AI accelerator kernels
Researchers have developed KForge, a framework that uses LLM-driven agents to automatically generate optimized kernels for AI accelerators. This system addresses the challenge of creating efficient code for diverse hard…
-
vLLM production guide details key config decisions for performance
This article provides a guide for optimizing vLLM deployments, focusing on three critical configuration decisions that impact performance and cost. It details how static KV cache allocation can lead to GPU out-of-memory…
-
Together AI introduces AutoJudge for faster LLM inference
Researchers at Together AI have developed AutoJudge, a novel method to accelerate large language model inference. This technique automates the curation of task-specific datasets, enabling lossy speculative decoding with…