提示缓存和KV缓存是高效LLM推理的关键优化，可显著降低延迟和成本。提示缓存会在设定的时间内存储对相同提示的响应，默认的五分钟TTL在新鲜度和效率之间取得平衡，在80%的命中率下可能将成本降低高达64%。KV缓存对于实时聊天至关重要，它存储先前token的键（Key）和值（Value）状态，将生成过程从二次方复杂度转变为线性复杂度，并支持更长的上下文窗口，尽管它会消耗大量GPU内存。 AI
影响 这些缓存技术对于使LLM应用程序在实时用例中更快、更便宜、更具可扩展性至关重要。
排序理由 该集群讨论了LLM推理的技术优化，特别是提示缓存和KV缓存，它们是该领域的核心研究课题。
AI 生成摘要 · Google Gemini · 来自 3 个来源。 我们如何撰写摘要 →
<h2> The Mechanics of Prompt Caching: Beyond the API Docs </h2> <p>Prompt caching is a critical technique for optimizing AI workflows, especially when dealing with repetitive or similar prompts. While API documentation often emphasizes the basic concept, storing responses to redu…
<p>Prompt engineering started as a narrow craft and then grew into a much bigger idea.</p> <p>At first, it just meant learning how to write better instructions so a model would give better answers. That is still part of the job. But once people started building real AI agents, th…
<p>The KV-cache is the single most important optimisation in LLM inference — and the reason real-time chat with a model is even feasible. Here's what it is and why it matters.</p> <h2> Generation is autoregressive </h2> <p>An LLM produces text one token at a time: emit a token, a…