Linear regression ICL

Transformer training methods impact circuit removability

A new research paper explores how training trajectories influence the removability of circuits in annealable soft-prior Transformers. The study found that specific training methods, like smooth fade-to-zero training, ar…