New TTT-NTP method boosts LLM performance using next-token prediction

Researchers have introduced a new method called Test-Time Training with Next-Token Prediction (TTT-NTP) that enhances the performance of pre-trained long-context language models. This technique leverages the inherent next-token prediction signal used in language model training to guide the adaptation process during test-time training. Unlike previous methods that used proxies, TTT-NTP directly supervises updates using the model's own contextual hidden states, ensuring that each adaptation step aligns with the core training objective. Evaluations on benchmarks like RULER Full-13 and LongBench-v2 show consistent improvements across various models, including Llama-3.1-8B and Mistral-7B-v0.3, without compromising existing capabilities. AI

IMPACT This method could improve the adaptability and performance of deployed LLMs by leveraging their core training objective.

RANK_REASON The cluster contains an academic paper detailing a new method for language model adaptation. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• Llama-3.1:8b
• LongBench-v2
• Mistral-7B-v0.3
• Qwen3 0.6B
• Qwen3-4B
• RULER Full-13
• Test-Time Training with Next-Token Prediction
• TTT-NTP
• Xuan Ouyang

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →