PulseAugur
中
实时 06:51:21
English(EN) TIDE: Temporal Incremental Draft Engine for Self-Improving LLM Inference

新研究探索跨不同硬件和架构的高效 LLM 推理技术 · 跟踪 10 个来源

多篇研究论文探索了优化大型语言模型 (LLM) 推理的新颖方法,重点关注效率和可访问性。CoMoE 和 Cascadia 旨在通过提高通信效率和驻留执行,在普通硬件上实现专家混合 (MoE) 模型。SchemaFill 和 BitNest 引入了投机解码技术,以加速 LLM 工具调用并减少内存开销。DySCo 和 EdgeAgent 通过优化 KV 缓存管理和统一内存架构上的动态调度,来解决边缘-云协同推理和多代理系统问题。最后,TopK-Guided 和 SparseEngine 专注于激活稀疏性和高效的 KV 缓存处理,以加快 LLM 推理速度并降低计算成本。 AI

影响 这些进展旨在使 LLM 在各种硬件上更易于访问和更高效,有可能加速其在边缘计算和专业应用中的采用。

排序理由 多篇 arXiv 论文提出了关于优化 LLM 推理效率和部署的新研究。

在 arXiv cs.AI 阅读 →

AI 生成摘要 · Google Gemini · 来自 36 个来源。 我们如何撰写摘要 →

新研究探索跨不同硬件和架构的高效 LLM 推理技术 · 跟踪 10 个来源

本文如何被排名

Signal score
0 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Research
多篇 arXiv 论文提出了关于优化 LLM 推理效率和部署的新研究。
Source corroboration
36 independent sources
Strong cross-source corroboration — multiple independent publishers covered this within the clustering window.
Topics
infra, model release
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
58 days old
Aged out of breaking-news scoring windows; ranking reflects the durable signal from the full source set.
Coverage growth since scoring
+4 source(s) since last score
New sources have picked up this story since our last re-score. Score will update on the next scoring pass.

完整方法见我们的编辑标准。

报道来源 [36]

  1. arXiv cs.LG TIER_1 English(EN) · Ruwen Fan (Jimmy), Yuezhi Zu (Jimmy), Junru Li (Jimmy), Qingda Hu (Jimmy), Xinjun (Jimmy), Yang, Jiwu Shu, Youyou Lu ·

    使用CoMoE在商用GPU上普及MoE推理

    arXiv:2610.09424v1 Announce Type: cross Abstract: Deploying Mixture-of-Experts (MoE) models relies heavily on Expert Parallelism, which generates intense inter-GPU communication. Consequently, state-of-the-art inference systems require high-bandwidth, P2P interconnects (e.g., NVL…

  2. arXiv cs.AI TIER_1 English(EN) · Jingpo Xu, Paul Joe Maliakel, Ivona Brandic, Shashikant Ilager ·

    DySCo:用于深度同步批处理的协作边缘-云 LLM 推理的动态分片

    arXiv:2610.08268v1 Announce Type: cross Abstract: Pervasive intelligent applications are increasingly deployed on mobile and Internet of Things (IoT) edge devices. Consequently, Large Language Models (LLMs) are increasingly used to support these applications. Yet, due to their hi…

  3. arXiv cs.AI TIER_1 Español(ES) · Tate Berenbaum (Not Community Labs Inc.), Matias Parij (Not Community Labs Inc.), Muthaiah Venkatachalam (Intel Corporation) ·

    Cascadia:Eleven AI PC 上的 Resident 975B MoE 推理

    arXiv:2610.07219v1 Announce Type: new Abstract: Mixture-of-experts models make nearly trillion-parameter capacity accessible with sparse per-token computation, provided that the serving system can distribute the weights and coordinate their execution. We present Cascadia's reside…

  4. arXiv cs.AI TIER_1 English(EN) · Zhi-Kai Chen, Song-Yan Li, De-Chuan Zhan, Han-Jia Ye ·

    SchemaFill:通过槽位并行推测解码实现高效 LLM 工具调用

    arXiv:2610.07086v1 Announce Type: cross Abstract: LLM agents interact with external systems by generating structured tool calls. Given a user request, conversational context, and a catalog of tool schemas, a tool-calling model must select tools and generate their arguments, poten…

  5. arXiv cs.AI TIER_1 English(EN) · Indranil Halder, Cengiz Pehlevan ·

    揭秘LLM-as-a-Judge:用于推理时扩展的分析可处理模型

    arXiv:2512.19905v3 Announce Type: replace-cross Abstract: Recent developments in large language models have shown advantages in reallocating a notable share of computational resource from training time to inference time. However, the principles behind inference time scaling are n…

  6. arXiv cs.AI TIER_1 English(EN) · Chence Yang, Ningxi Cheng, Arash Akbari, Qitao Tan, Qingchan Zhu, Ci Zhang, Changdi Yang, Yanzhi Wang, Wei Niu, Jinhui Wang, Jin Lu, Geng Yuan ·

    BitNest:位嵌套的推测解码,用于内存高效的大模型推理加速

    arXiv:2610.02800v1 Announce Type: new Abstract: Speculative decoding accelerates autoregressive generation by using a lightweight draft to propose multiple tokens for parallel verification. However, existing methods often require an additional draft model or weight representation…

  7. arXiv cs.LG TIER_1 English(EN) · Zhihui Gao, Tingjun Chen, Dirk Englund ·

    AIR-LLM:通过射频计算在无线电上广播 AI 权重,实现无内存边缘 LLM 推理

    arXiv:2610.00465v1 Announce Type: cross Abstract: Next-generation large language models (LLMs) are expanding from the cloud to ubiquitous edge devices. However, edge devices typically either lack the memory to store increasingly large LLM weights or, even with enough memory, spen…

  8. arXiv cs.MA (Multiagent) TIER_1 English(EN) · Jiangsu Du ·

    EdgeAgent:在 CPU-GPU 统一内存架构上为终端用户多智能体系统编排设备端 LLM 推理

    Emerging multi-agent LLMs demand privacy-preserving edge deployment, yet current inference systems struggle with these collaborative workflows. Specifically, the memory-bound decode phase causes severe bus contention on unified memory architectures (UMA), paralyzing naive CPU-GPU…

  9. arXiv cs.LG TIER_1 English(EN) · Jitai Hao, Quansheng Gu, Qiang Huang, Jun Yu ·

    SparseEngine: Sparse-First 推理引擎

    arXiv:2609.39068v1 Announce Type: new Abstract: Long-context LLM agents accumulate interaction histories that strain KV-cache memory and attention computation. Although sparse attention reduces these costs, heterogeneous cache representations and workflows hinder integration with…

  10. arXiv cs.AI TIER_1 English(EN) · Mukund Agarwalla, Chih-Jen Lin ·

    TopK-Guided:高效LLM推理的自适应、预算感知激活稀疏性

    arXiv:2610.01763v1 Announce Type: new Abstract: Activation sparsity speeds up large language model (LLM) inference by setting unimportant activations to zero so that the corresponding computations can be skipped. Existing training-free methods, however, make different trade-offs:…

  11. arXiv cs.AI TIER_1 English(EN) · Devleena Das, Rajeev Patwari, Vikram Kumar Bukka, Nithin Kumar Guggilla, Elliott Delaye, Ashish Sirasao ·

    MINCE:通过少模型蒙特卡洛校准缩小 LLM 评估数据集

    arXiv:2606.22826v2 Announce Type: replace Abstract: Evaluating LLMs across many model variants---quantized, fine-tuned, or deployment-specific---requires running large benchmarks repeatedly, a process that can take tens of hours per model on edge hardware such as NPUs. Existing s…

  12. arXiv cs.AI TIER_1 English(EN) · Ian Arawjo ·

    如何对LLM裁判进行统计分析并信任结果:使用evalstats进行小样本AI评估的校准推断

    arXiv:2609.35815v1 Announce Type: cross Abstract: Researchers across academia increasingly base significance claims on LLM judge scores and small-sample AI evaluations. Yet without well-calibrated confidence intervals (CIs), hypothesis tests, and judge-bias corrections, such clai…

  13. arXiv cs.AI TIER_1 English(EN) · Junxuan Li, Arko Mukherjee, Soumyabrata Pal ·

    LLM 裁判在稀疏重叠下的验证:从推理到设计

    arXiv:2609.31857v2 Announce Type: replace Abstract: Validating an LLM-as-a-judge requires estimating its agreement with humans, yet annotation budgets rarely allow every item to be multiply labeled. We prove that this overlap sparsity is the first-order determinant of wrong deplo…

  14. arXiv cs.LG TIER_1 English(EN) · Yufan Zhang, Sagnik Mukherjee, Hao Peng ·

    从 Fisher 视角理解 LLM 参数更新稀疏性

    arXiv:2609.36262v1 Announce Type: new Abstract: Recent studies have observed that parameter changes during language-model post-training can be concentrated in a small subset of coordinates. This phenomenon has been reported in reinforcement learning, on-policy distillation, and s…

  15. arXiv cs.AI TIER_1 English(EN) · Jiyoung Park, Hankyu Jang, Changseok Song, Wookeun Jung ·

    TIDE: 用于自改进LLM推理的时间增量草稿引擎

    arXiv:2602.05145v2 Announce Type: replace-cross Abstract: Speculative decoding can substantially accelerate LLM inference, but realizing its benefits in practice is challenging due to evolving workloads. We present TIDE (Temporal Incremental Draft Engine), a serving-engine-native…

  16. Together AI blog TIER_1 English(EN) ·

    通过 IBM Cloud 和 NVIDIA 扩展我们的企业推理能力

    Enterprises can now run open models at production scale on a dedicated B300 inference cluster, built by Together AI, IBM Cloud, and NVIDIA

  17. MarkTechPost TIER_1 English(EN) · Michal Sutter ·

    Architect推出Liquid Inference,一个用于LLM推理的实时拍卖

    <p>Architect Financial Technologies has launched Liquid Inference, an LLM router that runs a live auction for every request. Liquid Inference is an LLM inference marketplace from Architect where providers bid to serve each prompt. The buyer pays the lowest offer that meets its ru…

  18. Medium — MLOps tag TIER_1 English(EN) · Muharrem Bozkuş ·

    停止购买 GPU:通过更智能的路由和 KV 缓存管理来扩展 LLM 推理

    <div class="medium-feed-item"><p class="medium-feed-image"><a href="https://pub.towardsai.net/stop-buying-gpus-scale-llm-inference-with-smarter-routing-and-kv-cache-management-785c6be30911?source=rss------mlops-5"><img src="https://cdn-images-1.medium.com/max/1376/0*jcsdOfpvoxM0W…

  19. Medium — MLOps tag TIER_1 English(EN) · Harshit Dawar ·

    大型语言模型(LLM)推理如何工作?详细解释,包括所有阶段!

    <div class="medium-feed-item"><p class="medium-feed-image"><a href="https://harshitdawar.medium.com/how-does-llm-inference-work-explained-in-detail-including-all-its-phases-bafc98e8fe5f?source=rss------mlops-5"><img src="https://cdn-images-1.medium.com/max/2600/1*O92G2f-pdsPMmb46…

  20. Medium — MLOps tag TIER_1 English(EN) · Zeenatriaz ·

    MLOps终极指南:使用vLLM和AMD ROCm优化LLM推理

    <div class="medium-feed-item"><p class="medium-feed-snippet">A production-ready blueprint for high-throughput machine learning systems engineering, advanced hardware acceleration, and zero-trust&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/@zeenatriaz468/ult…

  21. dev.to — LLM tag TIER_1 English(EN) · AI Engineering Illustrated ·

    LLM推理服务如何工作:KV缓存、PagedAttention和连续批处理

    <p><em>Why serving a model is mostly a memory-management problem, and the operating-systems idea that fixed it</em></p> <p>Training a large language model gets the headlines, but <strong>inference</strong> is where most of the compute bill goes in production. Every chat message, …

  22. dev.to — LLM tag TIER_1 English(EN) · Pujun Lun ·

    为什么连续批处理让我的LLM推理引擎变慢了

    <p>I had been adding optimizations to <a href="https://github.com/lun0522/mini-vllm-rs" rel="noopener noreferrer">mini-vllm-rs</a>, my Rust LLM inference engine for macOS, including prefix caching and speculative decoding. Continuous batching looked like the next easy win. Proces…

  23. r/LocalLLaMA TIER_1 English(EN) · /u/pmttyji ·

    GitHub - google-ai-edge/ml-drift: GPU 加速的 AI/ML 推理

    <table> <tr><td> <a href="https://www.reddit.com/r/LocalLLaMA/comments/1x1owzm/github_googleaiedgemldrift_gpuaccelerated_aiml/"> <img alt="GitHub - google-ai-edge/ml-drift: GPU-Accelerated AI/ML Inference" src="https://external-preview.redd.it/Eo062QJBVC2cEX9vYowPuCDCT_Fwve3BvtdW…

  24. dev.to — LLM tag TIER_1 English(EN) · Alex Chen ·

    你的 7B 模型不需要 H100:LLM 推理的实用 GPU 选型指南

    <h1> Your 7B Model Doesn't Need an H100: A Practical GPU Sizing Guide for LLM Inference </h1> <p><em>Researched October 2026. Sizing rules from public docs and community practice, not my own benchmarks.</em></p> <p>The most expensive mistake in LLM inference isn't picking the wro…

  25. dev.to — LLM tag TIER_1 English(EN) · Narev Bot ·

    LLM定价变更:InferenceNet、OpenInference、Relace和Wafer

    <p>Model price changes detected for InferenceNet, OpenInference, Relace and Wafer. Details below.</p>

  26. r/LocalLLaMA TIER_1 English(EN) · /u/mikebmx1 ·

    Java vllm-like 框架声称在本地 NVIDIA GPU 推理时达到 llama.cpp 90% 的性能,方法是将 Java 编译为 CUDA 和 cuTile

    <!-- SC_OFF --><div class="md"><p>TornadoVM: The Java to CUDA engine:</p> <p><a href="https://github.com/beehive-lab/TornadoVM">https://github.com/beehive-lab/TornadoVM</a></p> <p>jitLLM: The inference engine:</p> <p><a href="https://github.com/beehive-lab/jitllm">https://github.…

  27. Mastodon — fosstodon.org TIER_1 English(EN) · [email protected] ·

    Architect 推出 Liquid Inference,一个用于 LLM 推理的实时拍卖平台,允许用户在几秒钟内竞标计算能力。该系统旨在

    Architect has launched Liquid Inference, a real-time auction platform for LLM inference that allows users to bid for compute capacity in seconds. The system aims to optimise GPU utilisation and reduce costs for AI workloads. https://www. marktechpost.com/2026/10/08/ar chitect-lau…

  28. dev.to — LLM tag TIER_1 English(EN) · cognitalk ·

    面向数十亿至数百亿参数大模型的消费级 GPU 推理:技术洞见

    <h1> 消费级显卡跑百-千亿大模型推理技术原理探讨 </h1> <h2> 引言 </h2> <p>Strata,dsv41-flash-offload,和较早的KTransformers(趋境科技)推出的技术都能够在消费级硬件上跑几百亿乃至千亿的大模型了,他们技术实现上是统一的技术吗?</p> <h2> 一句话结论 </h2> <p><strong>现在确实是多条完全不同的技术路线,没有统一;但未来会收敛到一套「通用异构内存调度 + 稀疏MoE原生架构 + KV压缩」的统一范式,不过不会变成单一引擎。</strong><br /> 你提到的两个项目本质…

  29. dev.to — LLM tag TIER_1 English(EN) · Aleksei Romanov ·

    突破极限:Qwen 3.8 27B 在 NVIDIA B300 上实现极端推理加速(100 至 10k+ tok/s)

    <p>Deploying a 27-billion parameter reasoning model like <strong>Qwen 3.8 27B</strong> on modern hardware presents a stark paradox. If you boot a default configuration on an NVIDIA B300 SXM6 GPU and send a solitary stream, you will measure roughly <strong>104 tokens per second</s…

  30. dev.to — LLM tag TIER_1 English(EN) · Alex Chen ·

    我比较了 2026 年的 5 个 GPU 云用于 LLM 推理——我发现了什么

    <h1> I Compared 5 GPU Clouds for LLM Inference in 2026 — Here's What I Found </h1> <p><em>Researched October 2026. Prices from public pricing pages and third-party trackers.</em></p> <p>Running LLMs in production gets expensive fast. I spent a week comparing GPU cloud providers t…

  31. r/LocalLLaMA TIER_1 English(EN) · /u/Distinct-Pie2389 ·

    LLM推理仪表板

    <table> <tr><td> <a href="https://www.reddit.com/r/LocalLLaMA/comments/1wxm2og/llm_inference_dashboard/"> <img alt="LLM Inference Dashboard" src="https://preview.redd.it/i0wcup9bphth1.jpg?width=140&amp;height=80&amp;auto=webp&amp;s=a268356e31d6b4f5e4aa5f55dbce03a1c5798caa" title=…

  32. r/LocalLLaMA TIER_1 English(EN) · /u/carteakey ·

    过拟合推理引擎的兴起

    <table> <tr><td> <a href="https://www.reddit.com/r/LocalLLaMA/comments/1wwu6zj/the_rise_of_overfit_inference_engines/"> <img alt="The Rise of Overfit Inference Engines" src="https://external-preview.redd.it/aNcJUYeEP8Q3f5LSIpNIZjjRXbxAsnMQfvsJVWEZJys.png?width=640&amp;crop=smart&…

  33. dev.to — LLM tag TIER_1 English(EN) · Saptarshi Paul ·

    使用 WebGPU 在浏览器中进行 LLM 推理:2026 年实地指南

    <p>Nearly every AI feature in a web app today is a round trip: the page collects text, sends it to a hosted model, and streams tokens back over the network. That design puts a per-token bill, a network hop, and someone else's data retention policy between your user and a text box…

  34. r/LocalLLaMA TIER_1 English(EN) · /u/Postmodern_Plunger ·

    面向小白的推理工程

    <!-- SC_OFF --><div class="md"><p>Hi all! I am a former SWE who has recently transitioned into inference engineering. I launched a side hustle a few months back and I've just taken it full time due to excessive demand. Its been such an opportunity because the people optimizing ru…

  35. dev.to — LLM tag TIER_1 English(EN) · Yuri Pocepaev ·

    一个模型,多种角色:无需训练更多模型即可实现 LLM 推理的专业化

    <p>I ran the same ten tool-selection tasks with four different tool catalogues against one shared model backend: <strong>40 requests in total</strong>.</p> <p>With five tools, the requests used <strong>6,208 input tokens</strong> in total. With fifty tools, they used <strong>38,2…

  36. Mastodon — mastodon.social TIER_1 English(EN) · adityahalderdev ·

    面向开发者的 Qwen3.8-Flash-Next 本地推理指南:硬件选型、本地验证、OpenAI 兼容客户端及性能基准测试

    A developer-first guide to Qwen3.8-Flash-Next local inference with Strata: hardware sizing, localhost verification, an OpenAI-compatible client, and benchmark caveats. Vendor and community numbers are labeled. https:// codereportglobal.indevs.in/art icles/qwen38-flash-next-local-…