Research paper reveals FFNs actively steer long-context retrieval

A new research paper explores the role of Feed-Forward Networks (FFNs) in long-context retrieval tasks, moving beyond their traditional view as parametric memories. The study demonstrates that FFNs actively influence the retrieval state by either pushing it towards or away from the correct answer, a function that is task-conditioned and layer-specific. Researchers found that FFNs can act as suppressors or amplifiers, with their roles changing based on the retrieval mode. A diagnostic tool based on the directional derivative of FFN writes effectively predicts this suppressive behavior, showing significant accuracy on models like Qwen2.5-7B and Qwen3.5-9B and improving retrieval margins when used to guide FFN scaling. AI

IMPACT Reveals a new mechanism for understanding and potentially improving long-context retrieval in LLMs by analyzing FFN behavior.

RANK_REASON Academic paper detailing novel findings on model architecture and behavior. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• Qwen2.5-7B
• Qwen3.5-9B

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →