A new paper published on arXiv identifies pooling operations and semantic shift as the primary drivers of embedding collapse in long text, rather than text length or attention mechanisms alone. The research establishes a theoretical framework demonstrating how contextual pooling inherently causes semantic dilution and spatial concentration of vectors. Experiments show that semantic shift is the main predictor of embedding concentration, and anisotropy is only detrimental when caused by significant semantic shifts, offering a new explanation for challenges in long-context retrieval. AI
IMPACT Provides a theoretical framework and experimental evidence to address fundamental challenges in long text embedding, potentially improving retrieval systems.
RANK_REASON Academic paper detailing theoretical and experimental findings on challenges in long text embedding. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →