English(EN)The result is a vicious cycle. Agent A pauses for a tool call, its cache gets evicted, the tool returns, the engine recomputes Agent A's whole history from scra
Together 的 ThunderAgent 优化 AI 推理，提高吞吐量并降低延迟 · 跟踪 9 个来源
作者PulseAugur 编辑部·[9 个来源]·
Together 开发了 ThunderAgent，这是一个开源推理优化工具，旨在解决代理工作流中的 KV 缓存颠簸问题。当代理任务在 GPU 密集型推理和等待外部工具之间交替时，会出现此问题，导致内存使用效率低下和性能下降。ThunderAgent 通过将代理工作流视为可调度程序来解决此问题，从而实现更智能的内存管理和路由到具有可用容量的节点。据报道，该工具提供了显著的性能改进，包括单节点吞吐量提高高达 2.5 倍，在高并发下 P50 延迟降低约 10 倍，并被 ICML 2026 评为 Spotlight 论文。
AI
It's a drop-in.
One program_id field and it plugs into your existing engine configs such as KV offloading + speculative decoding. Format-agnostic by design, with OpenAI chat completions supported today. Already adopted by SkyRL and NVIDIA Dynamo.
X — Together (inference / OSS)
TIER_1English(EN)·togethercompute·
Multi-node: near-linear scaling, 671 to 2,248 steps/min from 16 to 64 GPUs. And the lead widens with cluster size, from 1.79× at 2 nodes to 2.39× at 8. https://t.co/HS1I6p0KLP
X — Together (inference / OSS)
TIER_1English(EN)·togethercompute·
The numbers. Single 8×H100 node, batch 192: SGLang drops to 390 tok/s at 65s mean latency.
ThunderAgent holds 803 tok/s at 10.6s. That's 2× throughput and roughly 6× lower latency. https://t.co/X2sarlqYSL
X — Together (inference / OSS)
TIER_1English(EN)·togethercompute·
With that abstraction it does program-levels scheduling: under memory pressure it pauses low-priority workflows so the rest hit far higher cache-hit rates.
Resumed workflows route through a global queue to the node with the most free capacity.
X — Together (inference / OSS)
TIER_1English(EN)·togethercompute·
The root cause: request-level engines never see that a series of LLM calls belongs to one longer workflow.
ThunderAgent adds that missing view. It treats each agent workflow as a schedulable program, tracking its phase, KV footprint, and node placement. https://t.co/p5XBcrsiC5
X — Together (inference / OSS)
TIER_1English(EN)·togethercompute·
The obvious fixes only delay it. More nodes? Static pinning leaves some nodes starved, others idle. Offloading to CPU/disk? Expands capacity but thrashing returns once the working set outgrows every tier.
X — Together (inference / OSS)
TIER_1English(EN)·togethercompute·
The result is a vicious cycle. Agent A pauses for a tool call, its cache gets evicted, the tool returns, the engine recomputes Agent A's whole history from scratch, which evicts Agent C. Cascade.
This is KV cache thrashing. https://t.co/IHWzXNgRFV
X — Together (inference / OSS)
TIER_1English(EN)·togethercompute·
The problem: agent workflows alternate between GPU-heavy reasoning and GPU-idle waiting on tools. Run hundreds concurrently and their KV caches fight for memory.
Engines evict on a dumb LRU policy, even when that cache is needed again in seconds.
X — Together (inference / OSS)
TIER_1English(EN)·togethercompute·
Agentic inference wastes GPUs on KV cache thrashing.
ThunderAgent fixes it at the scheduler level: 2.5x higher single-node throughput and ~10x lower P50 latency at high concurrency.
ThunderAgent was accepted to ICML 2026 as a Spotlight paper.
Full deep-dive 👇 https://t.c…
