Transformer parameter space geometry hinders learning of sensitive functions

Researchers have identified a key reason why transformer models struggle to learn certain functions, such as PARITY. The study reveals that even when these functions are representable by transformers, the specific parameter settings required occupy an extremely small region of the parameter space. This makes it highly improbable for random initialization to discover these settings, effectively rendering such functions unlearnable by standard transformer architectures. AI

IMPACT Identifies a fundamental limitation in transformer architectures, potentially guiding future model design for improved learning capabilities.

RANK_REASON Academic paper detailing a theoretical limitation of transformer models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• PARITY
• Transformers

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →