研究人员正在开发新的多模态持续指令调优方法，以提高大型语言模型的效率和性能。一种名为CRAM的方法使用质心路由和自适应专家混合（MoE）来隔离特定任务的模式并有效分配参数，从而缓解灾难性遗忘。另一种方法ProtoAda采用原型引导的自适应调优，并结合了面向格式的任务原型来改进路由和参数整合。此外，一个名为PROXY-MIX的框架在一个小型代理模型上学习一个动态重放控制器，并将其转移到更大的模型上，以在持续调优过程中保留能力和对齐行为。 AI
影响 这些进展旨在通过提高多模态LLM在不遗忘先前任务的情况下学习新任务的能力，使其在实际应用中更具适应性和效率。
排序理由 多篇研究论文介绍了多模态持续指令调优的新颖方法。
AI 生成摘要 · Google Gemini · 来自 9 个来源。 我们如何撰写摘要 →
arXiv:2506.01850v2 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) have achieved remarkable success in instruction-following tasks by integrating pretrained visual encoders with large language models (LLMs). However, existing approaches often strug…
arXiv:2606.03871v1 Announce Type: cross Abstract: Visual instruction tuning effectively adapts a pre-trained Large Language Model (LLM) to process image information alongside text. Yet, it remains unclear how visual features are embedded into the layer-wise hierarchy of abstracti…
Visual instruction tuning effectively adapts a pre-trained Large Language Model (LLM) to process image information alongside text. Yet, it remains unclear how visual features are embedded into the layer-wise hierarchy of abstractions of the LLM backbone. Across a diverse set of v…
arXiv:2606.02502v1 Announce Type: new Abstract: Multimodal Large Language Models (MLLMs) unify heterogeneous vision-language tasks under a shared generative framework via instruction tuning, yet real-world deployment demands continuous capability expansion, making Multimodal Cont…
arXiv:2606.00400v1 Announce Type: new Abstract: Continual instruction tuning updates a language model through a sequence of new domains, yet each update can progressively erode previously learned capabilities and alignment behavior. Replay is the standard mitigation, but fixed re…
arXiv:2606.02576v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) achieve strong performance through instruction tuning, but real-world deployment requires them to continually acquire new vision-language capabilities, making Multimodal Continual Instructi…
Multimodal Large Language Models (MLLMs) achieve strong performance through instruction tuning, but real-world deployment requires them to continually acquire new vision-language capabilities, making Multimodal Continual Instruction Tuning (MCIT) essential. To reduce inter-task i…
Multimodal Large Language Models (MLLMs) unify heterogeneous vision-language tasks under a shared generative framework via instruction tuning, yet real-world deployment demands continuous capability expansion, making Multimodal Continual Instruction Tuning (MCIT) essential. Exist…
arXiv:2511.20158v2 Announce Type: replace Abstract: While continual visual instruction tuning (CVIT) has shown promise in adapting multimodal large language models (MLLMs), existing studies predominantly focus on models without safety alignment. This critical oversight ignores th…