MorletQK

FourierQK method boosts transformer attention via spectral preprocessing

Researchers have developed a novel method called FourierQK that significantly enhances transformer attention mechanisms by applying spectral preprocessing to query-key projections. This technique, particularly effective…