Hierarchical Concept Geometry in Language Models Emerges from Word Co-occurrence
A new research paper proposes a distributional theory explaining how hierarchical concepts, like the "is-a" relationship, are represented geometrically within language models. The study suggests that the spectral organization of word co-occurrence statistics naturally leads to a hierarchical splitting geometry in embeddings. This phenomenon was observed in word2vec embeddings and also extended to Gemma 2B unembeddings, indicating that complex conceptual hierarchies can emerge from basic statistical patterns rather than requiring specialized mechanisms. AI
IMPACT Explains how conceptual hierarchies in LLMs can emerge from statistical word patterns, potentially simplifying future model design.