Language models can learn during inference, research finds

A new research paper explores the concept of language models learning during inference, a process termed dynamic evaluation. The study found that standard softmax-attention models, when allowed to perform gradient steps on text chunks, showed a significant reduction in next-token loss and outperformed frozen models on long-context benchmarks. However, this adaptation primarily improved style and vocabulary recall rather than factual retrieval. For masked-diffusion models, the adaptation was more pronounced, leading to substantial gains. The research also demonstrated that by distilling a model's own conclusion-conditioned judgments into a low-rank adapter, performance on tasks like GSM8K could match in-context learning ceilings without accuracy loss. AI

IMPACT Suggests potential for models to improve performance on long-context tasks through in-inference learning, though factual recall remains a challenge.

RANK_REASON Research paper detailing novel findings on language model adaptation during inference. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• BABILong
• ContractNLI
• DiffusionGemma 26B
• Gemma 4 E4B
• Gradient Adaptation
• GSM8K
• HotpotQA
• Language Models
• LongBench-v2
• Musique
• Nemotron-Diffusion-8B
• Qwen3 1.7B
• Qwen3-4B
• Qwen3.5 4B
• Qwen3.8-27B
• ruler
• SGD

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →