tokens per second

LLM benchmarks mislead on inference speed for long contexts

Current LLM inference benchmarks are misleading because they primarily measure short-context performance, which does not reflect real-world usage involving longer contexts. This discrepancy arises from the differing com…