Attention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigation
A new survey paper published on arXiv details the phenomenon of "Attention Sink" in Transformer models. This issue causes the models to focus disproportionately on uninformative tokens, complicating interpretability and exacerbating problems like hallucinations. The paper categorizes existing research into utilization, interpretation, and mitigation strategies, aiming to guide future advancements in Transformer architectures. AI
IMPACT Highlights a persistent challenge in Transformer models that impacts interpretability and performance, guiding future research.