Word Class Representations Spontaneously Emerge from Successor Representations Trained on Natural Language
Researchers have applied successor representations (SRs), a principle from reinforcement learning, to natural language processing. By training a neural network on WikiText-103 to predict future word distributions across various time horizons, they observed the spontaneous emergence of structured language representations. These representations exhibit a clear geometric organization related to part-of-speech categories, with nouns, verbs, and adjectives becoming separable through unsupervised clustering. The study suggests that syntactic categories may arise as a natural consequence of predictive sequence learning, bridging concepts from reinforcement learning, linguistics, and cognitive neuroscience. AI
IMPACT Suggests syntactic categories may emerge from predictive learning, potentially influencing future language model architectures.