Toy Models of Superposition

MSE loss hinders superposition in neural networks, researchers find

Researchers have demonstrated that Mean Squared Error (MSE) loss is ineffective for training neural networks to encode features in superposition, a technique where more features are represented than there are neurons. T…