MorletQK

FourierQK technique boosts transformer attention with spectral preprocessing · 2 sources tracked

Researchers have developed a novel technique called FourierQK that significantly enhances transformer attention mechanisms by applying spectral preprocessing to query-key projections. This method, tested on character-le…