Depth-m Attention Factorization

Keyless Attention mechanism halves KV cache and boosts transformer efficiency

Researchers have introduced Keyless Attention, a novel attention mechanism for transformers that eliminates the key projection entirely, operating solely on queries and values. This approach results in a Value-Only Cach…