TraceRelay architecture improves recurrent neural network performance on sequence tasks

Researchers have introduced TraceRelay, a novel attention-aligned recurrent neural network architecture. This design distributes persistent representations across a sequence of low-dimensional traces, enabling local attention to form increments from lower-layer representations. The architecture uses a fixed additive phase recurrence to accumulate these delayed increments, with a stride-wise prefix sum facilitating parallel prefill and bounded-buffer continuation. Experiments on tasks like Equal Repeats and Dyck closing-type prediction demonstrated that models with recurrent phase inheritance achieved significantly higher accuracy compared to independently trained variants, especially at longer sequence lengths. AI

IMPACT Introduces a novel recurrent architecture that enhances performance on sequence modeling tasks, potentially influencing future model designs.

RANK_REASON The cluster contains a research paper detailing a new model architecture. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• alphaXiv
• arXiv
• CatalyzeX Code Finder for Papers
• Connected Papers
• DagsHub
• Equal Repeats
• Gotit.pub
• Hugging Face
• IArxiv
• Litmaps
• ScienceCast
• scite Smart Citations
• TraceRelay

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →