DIVE: Embedding Compression via Self-Limiting Gradient Updates
Researchers have developed DIVE, a new method for compressing high-dimensional embeddings from large language models to reduce storage and computational costs in vector search systems. DIVE employs a self-limiting triplet loss to prevent excessive perturbation of pretrained embeddings and a contrastive loss that treats multiple projections of an embedding as implicit views. This approach aims to overcome overfitting issues common in existing compression methods, especially when labeled data is scarce, and has demonstrated superior performance across multiple datasets compared to prior techniques. AI
IMPACT Reduces the computational and storage overhead of LLM embeddings, potentially enabling more efficient and scalable vector search applications.