On the Infinite Width and Depth Limits of Predictive Coding Networks
Researchers have theoretically analyzed the infinite width and depth limits of Predictive Coding Networks (PCNs), an alternative to standard backpropagation. Their findings indicate that for linear residual networks, PCNs can achieve the same gradient computations as backpropagation under specific parameterizations. This convergence to backpropagation's loss function occurs when the model's width significantly exceeds its depth, suggesting a potential for local updates in brain-like network architectures. AI
IMPACT Provides theoretical grounding for alternative training methods, potentially influencing future neural network architectures.