GMA Network

Gaussian Mixture Attention offers linear-time sequence mixing

Researchers have introduced Gaussian Mixture Attention (GMA), a novel sequence mixing technique designed to overcome the quadratic scaling bottleneck of standard Transformer attention. GMA replaces explicit token-to-tok…