Geometry-Preserving Unsupervised Alignment for Heterogeneous Foundation Models
Researchers have developed a new framework called GPUA to better align vision-only and vision-language foundation models. This method treats features from vision-only models as a visual language, learning a mapping to integrate them into the semantic space of vision-language models. The alignment process preserves geometric information and reduces the modality gap without requiring labels or model parameter updates. Experiments show improved cross-model compatibility and enhanced performance on downstream tasks like zero-shot recognition and segmentation. AI
IMPACT Enhances cross-model compatibility, potentially improving performance on various computer vision tasks.