Together's ThunderAgent optimizes AI inference, boosting throughput and reducing latency · 9 sources tracked

COVERAGE [9]

1. X — Together (inference / OSS) TIER_1 English(EN) · togethercompute · 2026-07-29 22:48

   It's a drop-in.

   It's a drop-in. One program_id field and it plugs into your existing engine configs such as KV offloading + speculative decoding. Format-agnostic by design, with OpenAI chat completions supported today. Already adopted by SkyRL and NVIDIA Dynamo.

2. X — Together (inference / OSS) TIER_1 English(EN) · togethercompute · 2026-07-29 22:48

   Multi-node: near-linear scaling, 671 to 2,248 steps/min from 16 to 64 GPUs. And the lead widens with cluster size, from 1.79× at 2 nodes to 2.39× at 8. https://

   Multi-node: near-linear scaling, 671 to 2,248 steps/min from 16 to 64 GPUs. And the lead widens with cluster size, from 1.79× at 2 nodes to 2.39× at 8. https://t.co/HS1I6p0KLP

3. X — Together (inference / OSS) TIER_1 English(EN) · togethercompute · 2026-07-29 22:48

   The numbers. Single 8×H100 node, batch 192: SGLang drops to 390 tok/s at 65s mean latency.

   The numbers. Single 8×H100 node, batch 192: SGLang drops to 390 tok/s at 65s mean latency. ThunderAgent holds 803 tok/s at 10.6s. That's 2× throughput and roughly 6× lower latency. https://t.co/X2sarlqYSL

4. X — Together (inference / OSS) TIER_1 English(EN) · togethercompute · 2026-07-29 22:48

   With that abstraction it does program-levels scheduling: under memory pressure it pauses low-priority workflows so the rest hit far higher cache-hit rates.

   With that abstraction it does program-levels scheduling: under memory pressure it pauses low-priority workflows so the rest hit far higher cache-hit rates. Resumed workflows route through a global queue to the node with the most free capacity.

5. X — Together (inference / OSS) TIER_1 English(EN) · togethercompute · 2026-07-29 22:48

   The root cause: request-level engines never see that a series of LLM calls belongs to one longer workflow.

   The root cause: request-level engines never see that a series of LLM calls belongs to one longer workflow. ThunderAgent adds that missing view. It treats each agent workflow as a schedulable program, tracking its phase, KV footprint, and node placement. https://t.co/p5XBcrsiC5

6. X — Together (inference / OSS) TIER_1 English(EN) · togethercompute · 2026-07-29 22:48

   The obvious fixes only delay it. More nodes? Static pinning leaves some nodes starved, others idle. Offloading to CPU/disk? Expands capacity but thrashing ret

   The obvious fixes only delay it. More nodes? Static pinning leaves some nodes starved, others idle. Offloading to CPU/disk? Expands capacity but thrashing returns once the working set outgrows every tier.

7. X — Together (inference / OSS) TIER_1 English(EN) · togethercompute · 2026-07-29 22:48

   The result is a vicious cycle. Agent A pauses for a tool call, its cache gets evicted, the tool returns, the engine recomputes Agent A's whole history from scra

   The result is a vicious cycle. Agent A pauses for a tool call, its cache gets evicted, the tool returns, the engine recomputes Agent A's whole history from scratch, which evicts Agent C. Cascade. This is KV cache thrashing. https://t.co/IHWzXNgRFV

8. X — Together (inference / OSS) TIER_1 English(EN) · togethercompute · 2026-07-29 22:48

   The problem: agent workflows alternate between GPU-heavy reasoning and GPU-idle waiting on tools. Run hundreds concurrently and their KV caches fight for memory

   The problem: agent workflows alternate between GPU-heavy reasoning and GPU-idle waiting on tools. Run hundreds concurrently and their KV caches fight for memory. Engines evict on a dumb LRU policy, even when that cache is needed again in seconds.

9. X — Together (inference / OSS) TIER_1 English(EN) · togethercompute · 2026-07-29 22:48

   Agentic inference wastes GPUs on KV cache thrashing.

   Agentic inference wastes GPUs on KV cache thrashing. ThunderAgent fixes it at the scheduler level: 2.5x higher single-node throughput and ~10x lower P50 latency at high concurrency. ThunderAgent was accepted to ICML 2026 as a Spotlight paper. Full deep-dive 👇 https://t.c…