研究人员开发了新的架构方法，以解决大型语言模型（LLM）在持续预训练和微调过程中出现的灾难性遗忘问题。其中一种方法TFGN引入了一个叠加层，可以在不改变核心Transformer的情况下实现参数高效更新，在不同领域和模型规模下均能显著保留先前的知识。另一种受生物视觉启发的UAM方法，采用双流架构将语义理解与动作控制分离，在VLA模型训练过程中保持多模态能力。这些进展旨在使模型能够持续学习，而不会降低先前获得的知识的性能。 AI
影响 LLM和VLA模型的新架构设计有望提高持续学习能力，减少微调和预训练过程中的知识退化。
排序理由 该集群包含两篇研究论文，介绍了LLM和VLA模型持续学习的新颖架构，解决了灾难性遗忘的问题。
AI 生成摘要 · Google Gemini · 来自 4 个来源。 我们如何撰写摘要 →
Continually pre-training a large language model on heterogeneous text domains, without replay or task labels, has remained an unsolved architectural problem at LLM scale. Existing methods rely on replay buffers, task identifiers, regularization penalties that scale poorly, or sen…
Vision--language--action (VLA) models are typically built by fine-tuning a pretrained vision--language model (VLM) on action data. However, we show that this standard recipe systematically erodes the VLM's multimodal competence, a side effect we call the embodiment tax. But do VL…
<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@zljdanceholic/stop-messing-with-the-loss-function-the-smarter-way-to-fix-catastrophic-forgetting-in-llms-423ea65eef25?source=rss------fine_tuning-5"><img src="https://cdn-images-1.medium.com/m…
<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/superagentic-ai/learning-fast-and-slow-whats-next-in-llm-fine-tuning-and-plastic-continual-learning-with-gepa-6ae53907d95e?source=rss------fine_tuning-5"><img src="https://cdn-images-1.medium.c…