WAV: Multi-Resolution Block Residual Routing for Deep Decoder-Only Transformers
Researchers have introduced WAV v1, a novel method for improving the training of deep decoder-only Transformers. This technique enhances residual routing by incorporating multi-resolution detail bases, which capture directional information about attention and MLP updates, as well as early versus late sublayer dynamics. WAV v1 demonstrates significant benefits in language modeling tasks like TinyStories and Text8, particularly at greater depths of 24 and 48 layers, outperforming existing methods with minimal parameter overhead. AI
IMPACT Introduces a novel routing mechanism that could improve the efficiency and performance of future large language models.