English(EN) When Does Sharing Work? Scaling a Shared SigLIP2 Across Image, Video, Audio, and Audio-Visual Representations

新的 Omni-Encoder 使用共享参数处理图像、视频、音频

作者 PulseAugur 编辑部 · [1 个来源] · 2026-11-05 05:00

研究人员开发了 Omni-Encoder，这是一种新颖的多模态编码器，它使用具有共享参数的单个 SigLIP-2 Vision Transformer 来处理图像、视频、音频和视听输入。这种方法与传统的依赖事后绑定或单独编码器的方法不同，它对所有模态进行端到端联合优化。该框架包括一个模态原生输入阶段和一个用于统一时空对齐的多模态旋转位置嵌入，而交替模态训练被证明对于稳定的联合优化至关重要。Omni-Encoder 使用单一参数集在所有四种模态的 16 个基准测试中均表现出竞争力，并且与单独训练相比，联合训练方法增强了每种模态。 AI

影响这项研究通过实现不同数据类型的联合训练，有可能带来更高效、更强大的多模态人工智能系统。

排序理由该集群包含一篇详细介绍新模型架构及其在基准测试中性能的学术论文。[lever_c_demoted from research: ic=1 ai=1.0]

在 arXiv cs.AI 阅读 →

AI 生成摘要 · Google Gemini · 来自 1 个来源。我们如何撰写摘要 →

本文如何被排名

Signal score: 27 / 100; Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket: Tool; 该集群包含一篇详细介绍新模型架构及其在基准测试中性能的学术论文。[lever_c_demoted from research: ic=1 ai=1.0]
Source corroboration: Single-source cluster; Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics: paper, model release; Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance: High; Clearly on-topic for AI-industry coverage.
Story freshness: Breaking (< 6h); Fresh story with cross-source coverage still developing. Ranking may shift as more sources report.

完整方法见我们的编辑标准。

报道来源 [1]

arXiv cs.AI TIER_1 English(EN) · Xiaofei Sun, Peng Wang, Huayao Liu, Xiaoxiao Xu, Jun Song, Deqing Li, Lin Qu · 2026-11-05 05:00

共享何时奏效？跨越图像、视频、音频和视听表示的共享 SigLIP2 的扩展

arXiv:2611.05406v1 Announce Type: cross Abstract: Contrastive pretraining has advanced image-text understanding, yet image, video, and audio encoders are still trained independently, with cross-modal alignment realized mainly through post-hoc binding to a frozen anchor space. We …

本文如何被排名

报道来源 [1]

共享何时奏效？跨越图像、视频、音频和视听表示的共享 SigLIP2 的扩展

相关实体

相关话题