Rank Collapse, Fixed Points, and the Renormalization Group Structure of MLP Residual Networks
Researchers have quantitatively demonstrated the analogy between deep neural network forward passes and renormalization group (RG) flows. Their study on MLP residual networks revealed that the effective rank of the residual stream decreases with depth, indicating a progressive integration of irrelevant data. This rank collapse was selective, depending on the input distribution's correlation length, and the network preserved only relevant degrees of freedom. The findings suggest that MLPs implement a selective coarse-graining procedure governed by the input's spectral structure, with most of the network operating near a fixed point. AI
IMPACT Provides a quantitative framework for understanding how MLPs process information, potentially guiding future architectural designs.