New framework TGO-II reveals how Vision Transformer representations evolve during training

Researchers have developed Transformer Geometry Observatory-II (TGO-II), a new framework for analyzing the geometric evolution of internal representations in Vision Transformers (ViTs) during supervised training. Using methods like Centered Kernel Alignment (CKA) and Singular Vector Canonical Correlation Analysis (SVCCA), TGO-II reveals that representational specialization increases across layers as training progresses. The framework also observed that intrinsic dimensionality grows before stabilizing, indicating an expansion of the representation manifold. Contrary to some hypotheses, token interaction structures remain strong throughout training, suggesting that representational complexity emerges through richer transformations rather than token decoupling. AI

IMPACT Provides new insights into the internal workings of Vision Transformers, potentially guiding future model development and interpretability efforts.

RANK_REASON The item is a research paper detailing a new framework and analysis of AI model representations. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• Singular Vector Canonical Correlation Analysis
• SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability
• TGO-II
• Transformer Geometry Observatory TGO-II
• Two-Nearest Neighbor Intrinsic Dimensionality
• TwoNN-ID
• Vision Transformers
• ViT-Small/16

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →