Self-training restructures LLMs, boosting surface markers while collapsing syntax

A new research paper proposes the Structural Depth Hypothesis (SDH) to explain how self-training restructures language models. The study found that while surface-level linguistic features like discourse markers increase, deeper syntactic structures such as questions and passives decline. This effect was observed across multiple models and architectures, suggesting it's a specific outcome of self-training rather than a general language model behavior. AI

Summary written by gemini-2.5-flash-lite from 1 source. How we write summaries →

IMPACT This research suggests that self-training may lead to LLMs that are superficially complex but lack deep syntactic understanding, impacting data curation and text detection.

RANK_REASON The cluster contains an academic paper detailing a new hypothesis about language model behavior. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• GPT-2
• Pythia
• Structural Depth Hypothesis

• paper
• model release