Researchers find modality gap in AI models can improve robustness

Researchers have investigated the modality gap in multi-modal models like CLIP, observing that images and texts often occupy separate distributions in the shared embedding space. This paper demonstrates that this gap can be beneficial for robustness, acting as a feature rather than a bug. By applying a simple post-processing technique to reduce the gap, the models' robustness to perturbations can be significantly increased without sacrificing clean accuracy. AI

IMPACT Suggests a method to improve the robustness of existing multi-modal models without performance degradation.

RANK_REASON Academic paper published on arXiv detailing findings about multi-modal model robustness.

Read on arXiv cs.CV →

• arXiv
• Rhea Chowers

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →