Paradoxical noise preference in RNNs
Researchers have discovered that recurrent neural networks (RNNs) can develop a paradoxical preference for noise during training. Contrary to the expectation that noise should be removed for optimal performance, these networks, particularly continuous-time RNNs, often perform best when some level of noise is present. This phenomenon is linked to noise-induced shifts in the network's internal dynamics, especially when noise is applied within the activation function, leading to a form of overfitting to the training noise itself. AI
IMPACT Reveals that training noise can become an integral part of learned computation, impacting the design of robust artificial RNNs.