Transformers can approximate Gaussian Kernel Regression, research shows

A new research paper demonstrates that standard softmax-attention transformers can approximate Gaussian kernel ridge regression (KRR) predictors during their forward pass. The study constructs a single-head transformer capable of implementing preconditioned Richardson iteration, a method for solving kernel systems. This work reveals a functional decomposition within transformers, where attention layers handle cross-token interactions and MLP layers manage intra-token arithmetic. Empirical tests on GPT-2 style transformers show progressive alignment with exact Gaussian KRR estimators across network depth. AI

IMPACT Demonstrates a theoretical link between transformer architectures and kernel methods, potentially informing future model design.

RANK_REASON Academic paper detailing theoretical and empirical findings on transformer architecture capabilities. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• Gaussian Kernel Regression
• GPT-2
• kriging
• Mingsong Yan
• multilayer perceptron
• Preconditioned Richardson Iteration
• softmax-attention transformer
• transformers

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →