研究人员正在开发能够处理和整合文本、音频和视频等各种数据类型的多模态大型语言模型（MLLM）。一种名为 MM-When2Speak 的方法侧重于通过预测何时应进行简短反应或完整回应来改进对话时序，性能提升三倍。其他研究则探索仅使用成对模态来训练 MLLM，以减少数据整理工作量，并通过自我蒸馏技术解决细粒度视觉理解的挑战。这些进展旨在创建更自然、更具吸引力、更强大的 AI 系统，使其能够更好地感知和与现实世界互动。 AI
影响 通过多样化的数据输入增强了 AI 理解和与现实世界互动能力，改善了对话参与度和细粒度感知。
排序理由 多篇研究论文详细介绍了多模态大型语言模型的新技术和方法。
AI 生成摘要 · Google Gemini · 来自 4 个来源。 我们如何撰写摘要 →
arXiv:2505.14654v2 Announce Type: replace-cross Abstract: Chatbots via large language models (LLMs) generate fluent responses but often struggle with when to speak, especially for brief, timely listener reactions during ongoing dialogue. We present a multimodal strategy for LLMs,…
Despite the impressive results achieved by multimodal large language models (MLLMs), their training typically relies on jointly curated multimodal data, requiring substantial human effort to construct multi-way aligned datasets and thereby limiting scalability across domains. In …
Multimodal Large Language Models (MLLMs) still struggle with fine-grained visual understanding, where answers often depend on small but decisive evidence in the full image. We observe a regional-to-global perception gap: the same MLLM answers fine-grained questions more accuratel…
AI leaders discussed multimodal systems, sensory computing, privacy risks, robotics, and future human-machine collaboration possibilities.