A new research paper explores how the architecture of Transformer feedforward blocks influences the preservation of rank across network depth during initialization. The study reinterprets skip connections and normalization layers not just as magnitude controllers, but as mechanisms for maintaining gradient rank. It suggests that skip connections help route gradients around rank-reducing components, while normalization placement manages a tradeoff between rank collapse and ensemble-like behavior. The paper also details how the two-matrix structure within Transformers, combined with width expansion and activation functions, preserves representation and Jacobian rank, with the width following a Marchenko-Pastur law. The initialization rank of the input-output Jacobian is shown to predict network training success on datasets like CIFAR-10. AI
IMPACT Provides theoretical insights into Transformer architecture, potentially guiding future model design for improved training stability and performance.
RANK_REASON The cluster contains a research paper detailing novel theoretical insights into Transformer architecture.
- alphaXiv
- CatalyzeX Code Finder for Papers
- CIFAR-10
- Connected Papers
- DagsHub
- Gotit.pub
- Hugging Face
- Influence Flower
- Litmaps
- Marchenko--Pastur law
- ScienceCast
- scite Smart Citations
- Transformer
AI-generated summary · Google Gemini · from 2 sources. How we write summaries →
