Analogies between Transformer Layers and Power Method
A new research paper proposes an analogy between the operations within a Transformer layer and the power method in numerical linear algebra. The paper demonstrates that tokens processed through a Transformer layer tend to align with the principal eigenvector of a specific matrix derived from the layer's weights. This alignment is particularly pronounced in Transformers with shared weights and suggests a method for directing the model's output. AI
IMPACT This theoretical finding could lead to new methods for understanding and controlling Transformer model behavior.