MLP skip connections can't be absorbed into residual-free models

Researchers have investigated whether a skip connection around a single-hidden-layer MLP can be absorbed into a residual-free MLP of the same width. They found that for certain activation functions like ReLU^2 and ReGLU, absorption is impossible due to degree arguments. For gated activations such as SwiGLU and GeGLU, a linearization argument also leads to the same conclusion. While absorption is possible for ungated ReLU and GELU under specific, non-generic weight conditions, skip-connected and residual-free MLPs generally represent distinct function classes. AI

IMPACT Explores theoretical limitations of MLP architectures, potentially influencing future model design.

RANK_REASON This is a research paper published on arXiv discussing theoretical properties of MLPs.

Read on arXiv cs.LG →

• GeGLU
• GELU
• ReLU
• SwiGLU

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →