Heo et al.

Cross-model KV cache sharing promises to speed up multi-model AI inference

Two research papers propose a method called cross-model KV cache sharing to improve the efficiency of multi-model AI inference pipelines. This technique allows the key-value states computed by one model during its initi…