Tencent's HiLS Attention Extends LLM Context to 4 Million Tokens

Tencent's Hunyuan team has developed a novel sparse attention mechanism called HiLS Attention, detailed in a new paper. This method makes chunk selection a learnable part of the model by using landmark tokens that summarize chunk content. This approach allows the model to maintain full-attention quality at in-domain lengths while extrapolating to contexts over 64 times longer than training, achieving a 13-15x speedup at 512K tokens. The HiLS Attention mechanism is available on Hugging Face, with open-sourced code. AI

IMPACT Enables LLMs to process significantly longer contexts, potentially improving performance on tasks requiring extensive information retrieval.

RANK_REASON Research paper detailing a new attention mechanism for LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on dev.to — LLM tag →

• arXiv:2607.02980
• HiLS Attention
• Hugging Face
• Hunyuan Model
• Tencent

• model release
• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →