Limitations of Learning Tanh Neural Networks with Finite Precision
A new paper explores the inherent limitations of training $\tanh$ neural networks using finite-precision computations. The research demonstrates that under such conditions, adaptive randomized algorithms are bound by the Monte Carlo convergence rate. This limitation persists unless the computational budget scales exponentially with network size, highlighting fundamental constraints on learnability for networks with localized bump functions. AI
IMPACT Highlights theoretical constraints on training efficiency for certain neural network architectures.