MLP Networks Show Quantitative Renormalization Group Structure

By PulseAugur Editorial · [3 sources] · 2026-06-09 02:19

Researchers have quantitatively demonstrated the analogy between deep neural network forward passes and renormalization group (RG) flows. Their study on MLP residual networks revealed that the effective rank of the residual stream decreases with depth, indicating a progressive integration of irrelevant data. This rank collapse was selective, depending on the input distribution's correlation length, and the network preserved only relevant degrees of freedom. The findings suggest that MLPs implement a selective coarse-graining procedure governed by the input's spectral structure, with most of the network operating near a fixed point. AI

IMPACT Provides a quantitative framework for understanding how MLPs process information, potentially guiding future architectural designs.

RANK_REASON This is a research paper detailing novel findings about the internal workings of MLP networks.

Read on Hugging Face Daily Papers →

paper
other

AI-generated summary · Google Gemini · from 3 sources. How we write summaries →

COVERAGE [3]

Hugging Face Daily Papers TIER_1 English(EN) · 2026-06-09 02:19

Rank Collapse, Fixed Points, and the Renormalization Group Structure of MLP Residual Networks

The analogy between deep neural network forward passes and renormalization group (RG) flows has been repeatedly noted in the literature, but existing treatments remain qualitative: depth is described as a coarse-graining scale, attention is likened to a partition function, and re…
arXiv stat.ML TIER_1 English(EN) · Parviz Haggi-Mani, Irina Rish · 2026-06-10 04:00

Rank Collapse, Fixed Points, and the Renormalization Group Structure of MLP Residual Networks

arXiv:2606.10324v1 Announce Type: cross Abstract: The analogy between deep neural network forward passes and renormalization group (RG) flows has been repeatedly noted in the literature, but existing treatments remain qualitative: depth is described as a coarse-graining scale, at…
arXiv stat.ML TIER_1 English(EN) · Irina Rish · 2026-06-09 02:19

Rank Collapse, Fixed Points, and the Renormalization Group Structure of MLP Residual Networks

The analogy between deep neural network forward passes and renormalization group (RG) flows has been repeatedly noted in the literature, but existing treatments remain qualitative: depth is described as a coarse-graining scale, attention is likened to a partition function, and re…

COVERAGE [3]

Rank Collapse, Fixed Points, and the Renormalization Group Structure of MLP Residual Networks

Rank Collapse, Fixed Points, and the Renormalization Group Structure of MLP Residual Networks

Rank Collapse, Fixed Points, and the Renormalization Group Structure of MLP Residual Networks

RELATED ENTITIES

RELATED TOPICS