Brief

Multi-Head Latent Attention (MLA) is a novel attention mechanism designed to significantly compress the KV cache in large language models. By projecting KV pairs into a low-dimensional latent space, MLA achieves substantial cache reduction, enabling models like DeepSeek-V2/V3 and Kimi K2.x to handle longer contexts and larger batch sizes with less memory. This technique alters how prefix caching and attention computations are implemented, offering a more efficient trade-off between memory usage and computational cost during model inference. AI

Researchers have developed a statistical theory that frames multi-head attention (MHA) as an ensemble of Nadaraya-Watson kernel regression estimators. This framework reveals that variance reduction in MHA is fundamentally tied to the decorrelation of outputs from different attention heads, rather than just the number of heads. They introduced the Head Diversity Index (HDI) to measure this decorrelation and derived an optimal head-dimension allocation strategy, suggesting a new architectural scaling law where optimal per-head dimension grows logarithmically with training set size. AI