Cosmopedia

Researchers explore growing Transformers with modular composition and layer-wise expansion

Researchers have explored a method for training Transformer models by incrementally adding new layers to a frozen base, maintaining a constant budget for trainable parameters. This approach, termed 'Growing Transformers…