New theory precisely characterizes Transformer length generalization on regular languages

Researchers have developed a new algebraic decomposition theory to precisely characterize which regular languages Transformer-based language models can generalize to longer sequences than they were trained on. This theory addresses limitations in classical Krohn-Rhodes decomposition theory, which is insufficient for understanding Transformer length generalization due to differences in basic building blocks. The new approach generalizes decomposition theory to infinite groups, enabling a polynomial-time decision algorithm for regular language membership. Experiments confirm the theory's accuracy in predicting Transformer behavior. AI

IMPACT Provides a theoretical foundation for understanding and improving Transformer length generalization capabilities.

RANK_REASON Academic paper detailing a new theoretical framework for AI model generalization. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• C*-RASP
• Hugging Face
• Krohn-Rhodes decomposition theory
• Regular languages of star height one
• syntactic monoid
• Transformer++

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →