New LLA technique slashes Transformer cache size, boosting capacity

Researchers have developed Looped Latent Attention (LLA), a novel cache compression technique for looped, weight-tied Transformers. LLA exploits the structured, low-rank nature of the Key/Value (K/V) cache across recurrence steps, storing compact latents instead of full K/V vectors. This method significantly increases batch capacity and reduces memory usage, enabling models like Ouro-1.4B to handle 768 sequences at 4k context with a 21.3x compression ratio on a single H200 GPU. LLA has demonstrated superior performance compared to other cache compression methods and shows promising results on models such as Huginn-3.5B. AI

IMPACT Reduces memory footprint for large language models, potentially enabling wider deployment and longer context windows.

RANK_REASON Academic paper detailing a new technical approach to transformer architecture. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• LLA
• Huginn-3.5B
• Looped Latent Attention
• Ouro-1.4B
• Transformers

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →