Mixture-of-Experts Transformers

SMELT architecture improves MoE Transformer training efficiency

Researchers have developed SMELT, a new architecture for Mixture-of-Experts (MoE) Transformers that improves training efficiency and downstream performance. By looping the middle layers of the transformer twice while ca…