Shikhar Srivastava

LayerRoPE method reinterprets Transformer norm growth as positional encoding

Researchers have introduced LayerRoPE, a novel method that reinterprets the growth of hidden state norms in Transformers as an emergent depth-positional encoding. Instead of suppressing this growth, LayerRoPE leverages …