Transformer Attention Geometry: Queries Expand, Keys Shrink

Researchers have investigated the geometric development of queries and keys within Transformer architectures during training. By training small GPT-like Transformers on character-level WikiText-103, they observed that query dimensions tend to expand while key dimensions shrink. This shrinking of keys results in a narrower spectrum of QK^T and more peaked attention weights. Further experiments confirmed a causal link, where restricting the key spectrum sharpens attention, while maintaining its initial dispersion softens it. AI

IMPACT Provides insights into the internal geometric dynamics of Transformer attention, potentially informing future model design and optimization.

RANK_REASON The item is a research paper detailing findings on Transformer attention mechanisms. [lever_c_demoted from research: ic=1 ai=1.0]

Read on Hugging Face Daily Papers →

• generative pre-trained transformer
• transformer
• WikiText-103

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →