First Passage Attention improves long-context language models

Researchers have introduced First Passage Attention (FPA), a novel mechanism designed to improve how language models handle long-range dependencies in text. Unlike traditional methods that sum contributions along a path, FPA uses path extrema to selectively erase intermediate context while retaining important older tokens. This approach, which generalizes the Forgetting Transformer (FoX), allows for more efficient memory management and better retrieval of specific information over extended sequences. Experiments show FPA outperforms comparable models in synthetic tasks and achieves perfect passkey retrieval even with a compressed key-value cache in a large language model. AI

IMPACT Introduces a novel attention mechanism that could improve long-context handling and information retrieval in LLMs.

RANK_REASON Academic paper introducing a new model architecture. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• arXiv
• First Passage Attention
• Forgetting Transformer
• Hugging Face

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →