Graph Transformers Avoid LLM-like "Attention Sinks" Due to Scoring Function

A new research paper explores the phenomenon of "attention sinks" in graph transformers, contrasting their behavior with large language models. Unlike LLMs, where attention often focuses on the first token, graph transformers like GRIT rarely exhibit this "attention sink" behavior on molecules. The study suggests that the symmetry and locality of graphs can limit attention, but on molecules, the attention scoring function itself is the primary factor preventing sinks. Replacing GRIT's gated logit with a dot product significantly increases sink rates, indicating that attention sinks are a property of dot-product attention rather than Transformers in general. AI

IMPACT This research clarifies how graph transformers process information, potentially leading to more efficient and effective models for molecular and graph-based data analysis.

RANK_REASON Research paper published on arXiv detailing novel findings about graph transformers. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• arXiv
• Hugging Face
• large language models
• molecules
• transformers

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →