English(EN) Our cache hit rate was 90 percent and the bill still climbed

开发者发现缓存命中率对LLM成本具有误导性

作者 PulseAugur 编辑部 · [1 个来源] · 2026-07-08 18:22

一位开发者尽管缓存命中率很高，却遇到了显著的成本超支，并发现该指标具有误导性。缓存有效地处理了大量廉价、重复的请求，但未能缓存少数但非常昂贵的长上下文查询。这种差异意味着，虽然90%的请求是从缓存提供的，但绝大部分实际成本仍然是由未缓存的高令牌数提示产生的。 AI

影响强调了在LLM部署中，除了简单的缓存命中率之外，具有成本意识的指标的重要性。

排序理由开发者分享了关于LLM成本管理的个人经验和教训。

AI 生成摘要 · Google Gemini · 来自 1 个来源。我们如何撰写摘要 →

报道来源 [1]

dev.to — LLM tag TIER_1 English(EN) · Jasmine Park · 2026-07-08 18:22

Our cache hit rate was 90 percent and the bill still climbed

<p>The dashboard said we were serving nine of every ten requests from cache. The invoice said we were paying more every week. Both numbers were correct. The hit rate was counting requests and the bill was counting dollars, and those are not the same thing once your traffic is lop…