New metric reveals how transformer models use depth

Researchers have introduced a new metric called 'effective depth' ($\Deff$) to analyze the residual stream in transformer language models. This metric quantifies how representation similarity decays with layer distance, providing a single scalar value. Across sixteen decoder-only models, $\Deff$ revealed that most models exhibit a calibrated signature of correlated residual updates rather than indicating unused depth. Notably, Qwen3.5 and OLMo-2 showed significantly lower effective depths compared to their theoretical maximums, suggesting a specific pattern in their residual stream processing. AI

IMPACT Introduces a novel diagnostic tool for understanding internal model dynamics, potentially aiding in future model development and analysis.

RANK_REASON Academic paper introducing a new metric for analyzing transformer models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• arXiv
• Hugging Face
• OLMo-2
• Pythia
• Pythia-1.4B
• Qwen 3.5

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →