New theory suggests transformers use geometric memorization

Researchers have proposed a new theory of how transformer language models memorize factual information, suggesting a 'geometric' form of memorization rather than traditional associative memory. This model posits that learned embeddings encode relational structure, with the MLP acting as a relation-conditioned selector. Experiments with a single-layer transformer demonstrated that logarithmic embedding dimensions suffice for memorizing random bijections, and the MLP learned a generic selection mechanism transferable to new facts. AI

影响 Proposes a new understanding of how LLMs store information, potentially leading to more efficient model architectures.

排序理由 Academic paper detailing a new theoretical model for transformer memorization. [lever_c_demoted from research: ic=1 ai=1.0]

在 arXiv cs.CL 阅读 →

• Transformers

• 论文
• 模型发布

AI 生成摘要 · Google Gemini · 来自 1 个来源。 我们如何撰写摘要 →