New HiCI module enhances LLaMA-2 context to 100K tokens

Researchers have developed a new attention module called HiCI (Hierarchical Construction--Integration) designed to improve long-context language modeling. This module explicitly structures information hierarchically, constructing segment-level representations, integrating them into a global context, and then broadcasting this context to condition attention. When applied to LLaMA-2 with minimal parameter additions, HiCI extended context windows to 100K and 64K tokens, showing consistent improvements across various benchmarks. AI

IMPACT This research offers a new architectural approach to extending context windows in LLMs, potentially improving performance on tasks requiring long-range understanding.

RANK_REASON The cluster describes a new research paper introducing a novel module for language models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• GPT-3.5-Turbo-16K
• LLaMA-2
• Xiangyu Zeng

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →