Researchers have developed a novel method called Split Forward Gradient (Split-FG) to train deep neural networks more efficiently by reducing memory usage. This technique splits a network into a trunk and an output head, calculating the head's gradient exactly while estimating the trunk's gradient using a Jacobian-vector product. This approach mitigates the noise in forward-mode gradients and avoids backpropagation through the trunk, leading to memory savings of up to 35% compared to traditional backpropagation. Experiments showed that Split-FG, when combined with a smaller learning rate for the trunk, achieved competitive results on benchmarks like WikiText-103 and CIFAR-10/100, though performance differences increase with trunk size. AI
IMPACT Reduces memory requirements for training large models, potentially enabling larger architectures or training on less powerful hardware.
RANK_REASON Academic paper introducing a new training method for deep neural networks. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
