Researchers have developed a new supervision objective called EXACT to improve long-context adaptation in language models. This method addresses a mismatch in packed training by assigning extra weight to targets that rely on longer effective contexts. Experiments on Qwen and LLaMA models demonstrated significant improvements in benchmarks like NoLiMa and RULER, particularly when evidence was located thousands of tokens away, while preserving performance on standard QA and reasoning tasks. AI
Summary written by gemini-2.5-flash-lite from 1 source. How we write summaries →
IMPACT Enhances language model ability to process and recall information from distant parts of long documents.
RANK_REASON The cluster contains an academic paper detailing a new method for improving language model performance. [lever_c_demoted from research: ic=1 ai=1.0]