Phase-Coherent Transformer advances complex-valued neural networks

Researchers have introduced the Phase-Coherent Transformer (PCT), a novel complex-valued architecture designed to better preserve phase information across layers. Unlike traditional softmax attention, PCT employs a smooth gate mechanism that replaces token competition with token-non-competing attention. In evaluations across various benchmarks, including long-range memory, hierarchical reasoning, and image classification, PCT demonstrated superior generalization and performance compared to standard and complex-valued Transformers, even on challenging tasks like LRA-Text. AI

IMPACT Introduces a new architectural principle for complex-valued transformers that may improve generalization and performance on various tasks.

RANK_REASON The cluster contains an academic paper detailing a new model architecture. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• arXiv
• Hugging Face
• Leona Hioki
• LRA-Text
• Phase-Coherent Transformer
• softmax transformer
• transformer

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →