PastAGI

LLM agents' million-token context costs analyzed: memory bandwidth is key

Generating tokens for long-context LLM agents incurs significant costs due to the computational demands of accessing cached tokens. The primary bottleneck is memory bandwidth, as models must read all previously cached t…