GPT-Neo

Researchers explore weight decay, in-context learning, and acceleration for Transformer models

Researchers have developed several new methods to improve the efficiency and theoretical understanding of Transformer models. One paper provides a functional-analytic characterization of weight decay, demonstrating its …