Successor representations reveal emergent word class structures in language models

Researchers have applied successor representations (SRs), a principle from reinforcement learning, to natural language processing. By training a neural network on WikiText-103 to predict future word distributions across various time horizons, they observed the spontaneous emergence of structured language representations. These representations exhibit a clear geometric organization related to part-of-speech categories, with nouns, verbs, and adjectives becoming separable through unsupervised clustering. The study suggests that syntactic categories may arise as a natural consequence of predictive sequence learning, bridging concepts from reinforcement learning, linguistics, and cognitive neuroscience. AI

IMPACT Suggests syntactic categories may emerge from predictive learning, potentially influencing future language model architectures.

RANK_REASON This is a research paper detailing a novel application of reinforcement learning principles to natural language processing, leading to emergent linguistic structures. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• Natural Language
• Part-of-speech categories
• Successor Representations
• WikiText-103

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →