Pretraining Recurrent Networks without Recurrence
Researchers have developed a new method called Supervised Memory Training (SMT) to pretrain recurrent neural networks (RNNs) without relying on traditional recurrence. SMT trains RNNs by reducing the process to supervised learning on memory transitions, bypassing the limitations of backpropagation through time. This approach enables time-parallel training and a stable gradient path, potentially allowing for more scalable RNN architectures capable of capturing long-range dependencies. AI
IMPACT Enables parallel training of RNNs, potentially unlocking scaling for models that build temporal abstractions.