Kasun Dewage

Spectral outliers in Transformer attention reveal dominant learned structures

Researchers have applied Marchenko-Pastur random matrix theory to analyze pre-trained transformer attention weights, identifying spectral outliers that represent dominant learned structures. By zeroing these identified …