MiniMax dropped a new attention architecture. [N]
MiniMax has introduced a novel attention architecture, MiniMax Sparse Attention (MSA), designed to handle context windows of up to 1 million tokens. This new approach restructures memory access patterns to avoid the quadratic complexity typically associated with long contexts, achieving significant speedups and reduced compute. MSA reportedly offers 4x faster execution than previous sparse attention methods, with per-token compute reduced to 1/20th at full context depth, and claims to be the first open-weight model with frontier coding, 1M context, and native multimodality. AI
![MiniMax dropped a new attention architecture. [N]](https://preview.redd.it/gvokff4l0z4h1.png?width=140&height=80&auto=webp&s=80f1c2fd94bf5a34696b868003146205b01a0316)
IMPACT Enables significantly longer context windows for AI models, potentially improving performance on tasks requiring extensive information recall.