Brief

Researchers have developed a new theory explaining the internal workings of deep transformers, viewing them as mean-field interacting systems that perform distributed inference. This theory introduces 'function vectors' as internal state representations that allow transformers to infer latent context variables at progressively finer scales through their layers. The research demonstrates that transformer depth and feedforward blocks enable more sophisticated in-context learning algorithms than previously understood. AI

Researchers have developed a new method for creating function vectors (FVs) to steer Large Language Models (LLMs) during in-context learning. The study explores variations in FV definitions, focusing on attention head selection and steering techniques. By employing gradient-based attributions with Layer-wise Relevance Propagation (LRP) for head selection and a distributed approach for steering, the method significantly enhances both efficiency and accuracy in guiding LLMs. AI