Brief

Researchers have identified "Rosetta Neurons," which appear to be universal across different neural network architectures. These neurons exhibit a sublinear power-law scaling relationship, meaning larger models contain more of them, but they constitute a smaller proportion of the total neurons. The study found that these Rosetta Neurons become more selective and specialized as models scale, and they can be effectively used to filter data for pre-training, achieving results comparable to oracle data filtering. AI