InfoNCE Induces Gaussian Distribution
Researchers have demonstrated that the InfoNCE contrastive learning objective inherently promotes a Gaussian distribution within learned representations. This finding was established through theoretical analysis under specific alignment and concentration assumptions, as well as through experiments on synthetic and CIFAR-10 datasets. The study suggests that this induced Gaussian structure offers a principled way to analyze and apply learned representations in various contrastive learning applications. AI
IMPACT Provides a theoretical framework for understanding representations learned via contrastive learning, potentially aiding in the development of more robust foundation models.