Zohra

New research reveals learning rate transfer in hybrid Transformer-SSM models

A new research paper explores learning rate (LR) scaling for hybrid architectures that combine Transformer and State-Space Model (SSM) blocks, which are increasingly used in production language models. The study found t…