WinoGrande

Prism Transformer introduces progressive head schedules for hierarchical attention

Researchers have introduced the Prism Transformer, a novel architecture that modifies the standard multi-head attention mechanism. Instead of allocating equal dimensional space to each attention head at every layer, Pri…