AV-JEPA model advances audio-visual self-supervised learning

Researchers have introduced AV-JEPA, a new self-supervised learning model that extends LeJEPA to handle both audio and visual data. This model utilizes an early-fusion Vision Transformer and modality dropout for masking, aiming to align embeddings from global and local views. AV-JEPA achieves strong classification performance on datasets like VGGSound and AudioSet, and offers out-of-the-box zero-shot audio-video retrieval capabilities without requiring complex components like decoders or contrastive negatives. AI

IMPACT Advances audio-visual self-supervised learning and zero-shot retrieval capabilities.

RANK_REASON The cluster contains a research paper detailing a new model architecture and its performance on benchmarks. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• AudioSet
• AV-JEPA
• LeJEPA
• SIGReg
• VGGSound
• vision transformer

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →