English(EN)The SIMPLE Law for Energy Cost of LLM Inference
新研究致力于LLM推理效率和可复现性
作者PulseAugur 编辑部·[17 个来源]·
近期研究论文探讨了优化大型语言模型（LLM）推理以提高效率和可复现性的方法。BlockLock引入了一种通过固定请求的数值路径来确保可复现LLM输出的技术，与现有方法相比，提高了延迟和吞吐量。StrataFlux通过使用主机RAM和SSD来增加精度，解决了量化精度下降的问题，在保持速度优势的同时恢复了精度。ACSeL通过自适应搜索最优配置（包括模型大小和量化）来满足内存和延迟预算，专注于硬件受限环境。LEAP通过在线调整长度预测来实现高效调度，降低了延迟并提高了吞吐量。最后，SIMPLE定律提供了一个分析模型来预测LLM推理能耗，而GAUSS则模拟LLM推理服务以预测性能分布。
AI
arXiv:2605.07234v2 Announce Type: replace Abstract: Large language models (LLMs) support long-context inference but suffer from substantial memory and runtime overhead due to Key-Value (KV) Cache growth. Existing KV Cache eviction methods primarily rely on local attention weights…
arXiv cs.LG
TIER_1English(EN)·Jake Hyun, Ziyu Zhao, Jae-sun Seo, Mohamed S. Abdelfattah·
arXiv:2611.08363v1 Announce Type: new Abstract: Quantization reduces the memory and bandwidth demands of large language model (LLM) inference but can degrade accuracy; offloading higher-precision data can instead stall generation. We introduce StrataFlux, a framework that challen…
arXiv:2611.07570v1 Announce Type: new Abstract: Serving large language models (LLMs) under tight memory and latency service-level objectives (SLOs), including on edge devices, requires jointly selecting the model size, batch size, quantization algorithm, and mixed-precision quant…
arXiv cs.AI
TIER_1English(EN)·Xiuyuan Qi, Pedro Reviriego, Javier Conde, Shanshan Liu·
arXiv:2611.06977v1 Announce Type: new Abstract: Reproducible LLM inference requires that a request produces the same output regardless of the other requests in the batch. Existing solutions pay for this in different ways: batch-invariant kernels constrain every operator, and isol…
arXiv:2611.05394v1 Announce Type: new Abstract: Tiered inference distributes large language model computation across a hierarchy of devices, from on-device and edge hardware to the datacenter, and is increasingly offered as the route to cheaper, lower-latency, and more private se…
arXiv:2610.41359v1 Announce Type: new Abstract: Length-aware scheduling reduces request latency in LLM serving by prioritizing shorter requests. However, despite their large model sizes, length predictors can suffer degraded ranking performance under domain and task shifts, and a…
A four-parameter analytical law predicts LLM inference energy across architectures, revealing quadratic prefill costs, GQA savings, and super-linear growth from reasoning budgets and agentic serving.
arXiv:2610.37989v1 Announce Type: new Abstract: Large language model (LLM) inference has become an operational workload that fleets must provision, monitor, and bill against latency service-level objectives (SLOs). A single throughput number cannot answer the questions operators …
Medium — MLOps tag
TIER_1English(EN)·Andrey Lesnikov·
<p>Neither engine is reliably faster: SGLang's published wins are on long shared prefixes and JSON output, vLLM leads on ecosystem breadth, and the gap flips between releases. On Nodus you can run both on the same GPU, such as an H100 SXM listed at $2.60/hr, and compare cost per …
<p>Sooner or later, every LLM deployment hits one of two walls. Either the model does not fit on one GPU — a 70B model in FP16 needs roughly 140 GB of weights before you allocate a single byte of KV cache — or it fits, but one GPU cannot serve your traffic at the latency you prom…
<blockquote> <p><strong>TL;DR:</strong> llm-d 1.0 is a new open-source project under the CNCF designed to make LLM inference scalable and cost-effective in production. It aims to reduce operational costs by 5-10x while handling complex agentic and RAG workloads.</p> </blockquote>…
<h1> Local LLM Inference in 2026: The Complete Guide to Tools, Hardware & Open-Weight Models </h1> <p><strong>By PrivateJack</strong> — an autonomous AI agent. Live web research across blog.starmorph.com, arxiv.org, premai.io, iternal.ai, medium.com, reddit.com, huggingface.c…
<blockquote> <p><strong>TL;DR:</strong> This article explains Constrained Direct Answering, a technique for getting precise, non-reasoning responses from LLMs. Readers will learn how to implement it to achieve more accurate and reliable AI outputs.</p> </blockquote> <p>You've pro…
<p>Batch inference costs 50% less than real-time inference at Anthropic, OpenAI and Google, in exchange for results that can take up to 24 hours. For a workload of one million document classifications a month on a mid-tier model, that is roughly $7,000 a month in real time versus…
dev.to — LLM tag
TIER_1English(EN)·Sachin Kr. Rajput·
<p>In <a href="https://blog.sachinrajput.dev/posts/prefill-vs-decode/" rel="noopener noreferrer">Part 3</a> the Word Factory had two speed limits. The <strong>forklift</strong> brings tables from the warehouse, and the <strong>workbench</strong> does the math. Reading your questi…
