研究人员发现，思维链（CoT）微调虽然提高了推理能力，但会显著降低混合线性注意力模型中的长上下文记忆。这个问题被称为“注意力遗忘”，会导致在“海量信息找针”（Needle-In-A-Haystack）等任务上的性能下降。一种名为QK-Restore的无训练新方法被提出，通过恢复微调前检查点的特定查询-键投影权重来解决此问题，成功恢复了长上下文能力，而没有牺牲推理性能。 AI
影响 解决了LLM微调中的一个关键问题，有望为高级推理任务带来更强大的长上下文能力。
排序理由 该集群包含一篇研究论文，详细介绍了一种解决LLM微调中特定问题的新方法。
在 Hugging Face Daily Papers 阅读 →
AI 生成摘要 · Google Gemini · 来自 4 个来源。 我们如何撰写摘要 →
arXiv:2606.11052v1 Announce Type: new Abstract: Chain-of-thought (CoT) supervised fine-tuning (SFT) is widely adopted to improve reasoning ability, yet we find that it systematically degrades long-context recall in hybrid linear-attention models. Across architectures including Hy…
Chain-of-thought (CoT) supervised fine-tuning (SFT) is widely adopted to improve reasoning ability, yet we find that it systematically degrades long-context recall in hybrid linear-attention models. Across architectures including HypeNet and Jet-Nemotron, retrieval performance on…
Chain-of-thought supervised fine-tuning degrades long-context recall in hybrid linear-attention models by biasing attention gradients toward short-range patterns, but a training-free method called QK-Restore can restore long-context capabilities by reverting query-key projections…
<!-- SC_OFF --><div class="md"><p>Surprised there's no real tooling for this given how much research exists on continual learning. </p> <p>Built pyrecall to fill the gap. Snapshots skill scores before/after fine-tuning, flags regressions, rolls back LoRA adapters by name. </p> <p…