Adrian Goldwaser

New theory maps transformer context to MLP weight patches

Researchers have demonstrated that the impact of context in transformer models can be precisely mapped to rank-1 patches on their MLP weight matrices and RMSNorm scale. This theoretical framework, which applies to moder…