Recurrent Neural Networks Exhibit Task-Specific Redundancy in Weight Space

Researchers have explored the functional redundancy within the weight space of recurrent neural networks, specifically using ordered real Schur coordinates in one-layer tanh RNNs. This method separates spectral blocks from nonnormal couplings, allowing for structured ablations while keeping input and readout maps constant. In a fixed-length copy task, certain nonnormal Schur couplings could be removed with minimal impact on performance, while others were crucial for accurate autonomous replay. The study found that the profile of loss-preserving ablations varies across different tasks and trained solutions, indicating approximate functional invariances rather than universal symmetries in recurrent weight space. AI

RANK_REASON Academic paper detailing a novel method for analyzing recurrent neural network weight spaces. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• alphaXiv
• arXiv
• CatalyzeX
• context-dependent integration
• Copy task performance and urinary incontinence in Alzheimer's disease
• DagsHub
• Flip-Flop
• Gotit.pub
• Hugging Face
• tanh
• IArxiv
• Influence Flower
• Schur
• ScienceCast
• sine generation

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →