Recursive Transformers show improved performance on limited data

Researchers have developed recursive Transformers designed to perform better under limited data conditions compared to standard Transformers. These new models reuse shared blocks and employ factorized embeddings to optimize parameter usage and reduce vocabulary-map parameters. Experiments across various data budgets and corpora demonstrated that recursive Transformers outperform standard models when trained on 10M to 100M words, remaining competitive with top performers in the BabyLM Challenge 2025. AI

IMPACT Offers a more efficient approach to training language models with limited datasets, potentially improving performance in data-scarce environments.

RANK_REASON Academic paper detailing a new model architecture. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• alphaXiv
• arXiv
• BabyLM Challenge 2025
• CatalyzeX
• CORE Recommender
• DagsHub
• Gotit.pub
• Hugging Face
• Recursive Transformers
• ScienceCast
• Transformers

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →