New research questions superposition in language models

A new paper analyzes the phenomenon of "superposition" in language models, where multiple solutions might be maintained simultaneously within a single representation. Researchers investigated this using three different training regimes: training-free, fine-tuned, and from-scratch. Their findings indicate that only models trained entirely from scratch demonstrated signs of utilizing superposition. In contrast, models in the training-free and fine-tuned regimes either collapsed superposition or did not use it, instead opting for shortcut solutions. AI

IMPACT This research clarifies conditions under which language models may or may not leverage superposition for complex reasoning tasks.

RANK_REASON The cluster contains an academic paper analyzing a specific phenomenon in language models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• alphaXiv
• arXiv
• CatalyzeX
• CORE Recommender
• DagsHub
• Gotit.pub
• Hugging Face
• Influence Flower
• Latent CoT
• Logit Lens
• Michael Rizvi
• ScienceCast

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →