Normalized gradient descent

New research explains how transformers perform in-context learning via gradient descent

Two new arXiv papers explore the theoretical underpinnings of in-context learning (ICL) in transformers. One paper demonstrates how transformers can perform in-context logistic regression by implicitly executing normali…