New framework enhances federated cross-modal retrieval with missing modalities

Researchers have developed RCSR, a new framework designed to improve federated cross-modal retrieval, particularly when dealing with data heterogeneity and missing modalities across clients. The system utilizes a frozen CLIP backbone, incorporating shared adapters for global knowledge transfer and optional client-specific adapters for personalization. RCSR employs prototype anchoring to help unimodal clients align with global semantics and a semantic router on the server to dynamically adjust aggregation weights, enhancing both overall retrieval accuracy and training stability. AI

IMPACT Improves cross-modal retrieval accuracy and stability in federated learning scenarios with heterogeneous and incomplete data.

RANK_REASON This is a research paper detailing a new framework for federated cross-modal retrieval.

Read on arXiv cs.CV →

• arXiv
• Flickr30K
• MS-COCO

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →