Tencent's Hunyuan team has developed a novel sparse attention mechanism called HiLS Attention, detailed in a new paper. This method makes chunk selection a learnable part of the model by using landmark tokens that summarize chunk content. This approach allows the model to maintain full-attention quality at in-domain lengths while extrapolating to contexts over 64 times longer than training, achieving a 13-15x speedup at 512K tokens. The HiLS Attention mechanism is available on Hugging Face, with open-sourced code. AI
IMPACT Enables LLMs to process significantly longer contexts, potentially improving performance on tasks requiring extensive information retrieval.
RANK_REASON Research paper detailing a new attention mechanism for LLMs. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
