Principles of Concept Representation in Sentence Encoders
Researchers have published a paper detailing four principles that govern how sentence encoders represent concepts. The study, which trained encoders on millions of word pairs, found that fine-tuning primarily recalibrates existing latent geometry rather than expanding it. Semantic information is concentrated in the final transformer layer, making cross-layer pooling unnecessary. The research also introduces two new evaluation datasets for assessing semantic gaps and paraphrasing. AI
IMPACT Provides foundational insights into sentence encoder capabilities, potentially guiding future model development and evaluation.