Structural Correspondence and Universal Approximation in Diagonal plus Low-Rank Neural Networks
Researchers have introduced a new framework called Structural Correspondence for neural networks that use parameter-efficient low-rank structures. This framework demonstrates that augmenting low-rank layers with a minimal sparse diagonal component, forming a Diagonal plus Low-Rank (DLoR) structure, is sufficient to achieve Universal Approximation. The study proves that DLoR components can reconstruct any full-rank transformation and restore the Universal Approximation Theorem for general activation functions, challenging the necessity of dense matrices for universal expressivity. AI
IMPACT Introduces a theoretical framework that could lead to more parameter-efficient neural network architectures.