80 GiB

LLM Decode Speed: Bandwidth and KV Cache Limit Performance on H100

A technical analysis reveals that for large language models running on hardware like the NVIDIA H100, decode speed at batch size one is primarily determined by memory bandwidth rather than compute power. The "ridge batc…