New 'multiple-descent' phenomenon observed in LSTM networks

Researchers have identified a novel 'multiple-descent' phenomenon in long short-term memory (LSTM) networks, where performance fluctuates through repeated up and down cycles after overtraining. Analysis indicates these performance cycles are linked to phase transitions between order and chaos in the model. Optimal training points are consistently found at the critical transition between these phases, with the best model performance typically occurring at the initial transition from order to chaos, where the 'edge of chaos' is widest, facilitating better exploration of weight configurations. AI

IMPACT This research reveals a new dynamic in neural network training, potentially offering insights into optimizing model performance and stability.

RANK_REASON The cluster contains an academic paper detailing a new phenomenon observed in deep learning models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• Ling Feng
• long short-term memory

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →