Researchers have developed a method to accelerate long-context decoding in large-scale hybrid LLMs without requiring retraining. This technique, called combined sparse attention, replaces most full-attention layers with linear-attention layers and then further optimizes the remaining full-attention layers. By attending to a sliding window of recent tokens, a pinned prefix, and a fixed budget of the most relevant tokens, the models achieve significant speedups. This approach has been shown to preserve accuracy across six hybrid models, ranging from 4B to 397B parameters, and demonstrates up to 2.22x faster decoding for 256k tokens on specific models. AI
IMPACT This method could significantly reduce the computational cost and time required for LLMs to process and generate long sequences of text.
RANK_REASON Research paper detailing a new method for accelerating LLM decoding. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
