SemiAnalysis 强调了 vLLM 团队所做的重大优化，特别是针对代理工作负载。这些改进包括一个新的 AgentX 基准测试，旨在发现长上下文、多轮任务中的问题。这些优化已使 Kimi 推理的高并发吞吐量提高了 6 倍以上，并通过允许在不遗忘状态的情况下进行跨机器服务拆分，提高了效率。 AI
影响 vLLM 的这些优化可能会提高 AI 代理的效率和性能，尤其是在处理长上下文和多轮交互方面。
排序理由 该集群详细介绍了由知名实体 (vLLM) 为代理工作负载开发的特定技术优化和新基准测试。
AI 生成摘要 · Google Gemini · 来自 5 个来源。 我们如何撰写摘要 →
完整方法见我们的编辑标准。
@vllm_project Our new AgentX benchmark helped uncover issues that only surface during multi-turn long-context tasks, and provided a real-world benchmark to hillclimb. Read more in our AgentX article (5/5) https://t.co/PgtKNYXdFr
@vllm_project With these and other optimizations, the team at vLLM brought Kimi inference to new heights, increasing high-concurrency throughput by more than 6x in 2 weeks. (4/5) https://t.co/BKTTjalmOy
@vllm_project Another optimization lets split-across-machines serving hand over a hybrid model's complete state, so the decode half doesn't start with amnesia. All of this came from replaying real agent traffic. (3/5) https://t.co/vGiOtQDiFl
@vllm_project Agents reread their whole history every turn. Reuse the model's cache of it and you skip that work. Saving that cache off-GPU was wasteful: agents each saved their own copy, and every turn re-saved everything from scratch. Now, one save per shared prefix, and each …
Shoutout to the cracked team at @vllm_project that implemented recent agentic workload optimizations. (1/5)🧵 https://t.co/BsXCtLBwvU