L0H0

Transformer attention relevance tied to data structure, study finds

Researchers have analyzed the Transformer's attention mechanism using Wilsonian renormalization group theory, treating it as a perturbation of a trained MLP fixed point. Their findings indicate that attention's relevanc…