New Tensor Framework Analyzes Transformer MLP Design Choices

Researchers have introduced CP-MLPs, a novel tensor framework designed to analyze the design choices within transformer feed-forward blocks. This framework distinguishes between tied and untied parameter sharing in multiplicative units, revealing that untied blocks offer greater width efficiency for certain types of interactions. Experiments with trained gated MLPs indicate that untied blocks maintain an advantage when considering matched input-projection budgets, though the precise benefits depend on the target structure. AI

IMPACT This research provides a theoretical lens for understanding and potentially optimizing the design of transformer feed-forward networks.

RANK_REASON The item is an academic paper detailing a new theoretical framework and experimental results for analyzing neural network architectures. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• alphaXiv
• arXiv
• CatalyzeX Code Finder for Papers
• Connected Papers
• CORE Recommender
• CP-MLPs
• DagsHub
• Gotit.pub
• Hugging Face
• IArxiv Recommender
• Litmaps
• ReLU$^2$
• Representation Guided Parameter Efficient Llm Unlearning
• ScienceCast
• scite Smart Citations
• SwiGLU
• transformer

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →