Transformer model conversion: Wiring knowledge transfer between sizes

A new research paper explores the transferability of knowledge between different sizes of Transformer models, specifically focusing on converting a 1.4 billion parameter model to a 410 million parameter version within the Pythia family. The study found that while representations align strongly, direct parameter conversion is destructive due to structural differences. The research proposes a method combining least-squares compensation and variance-preserving rescaling to efficiently transfer knowledge, achieving comparable performance with significantly fewer tokens compared to training from scratch, especially at lower budgets. The paper also identifies limitations at larger scale conversions, suggesting dimension-aware regularization as a potential solution. AI

IMPACT Provides insights into efficient knowledge transfer between LLM sizes, potentially reducing training costs and time.

RANK_REASON Academic paper detailing novel research findings on model conversion. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• arXiv
• Pythia
• Ravi Satya Durga Prasad Yenugula
• Transformer

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →