LSTM-Style Transformers Reduce Parameters and Cache Needs

Researchers have developed an LSTM-style transformer that reuses weights and shares a key-value cache across its recurrent core, significantly reducing parameter count and cache requirements. This approach allows for deep computation with fewer parameters and less memory, particularly beneficial in scenarios with limited data and model storage. The training method addresses a mismatch between parallel and recursive cache entries using a consistency loss, improving performance on disjoint documents and across various model widths. AI

IMPACT This research could lead to more efficient transformer models, reducing computational and memory requirements for deep learning tasks.

RANK_REASON Academic paper detailing a novel model architecture. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• alphaXiv
• arXivLabs
• CatalyzeX Code Finder for Papers
• CORE Recommender
• DagsHub
• FineWeb
• Gotit.pub
• Hugging Face
• IArxiv Recommender
• KV caches
• long short-term memory
• ScienceCast
• transformers

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →