Transformers Learn the Mestre-Nagao Heuristic
Researchers have trained a two-layer transformer encoder to classify rational elliptic curves based on their rank, achieving over 99% accuracy using the first 128 normalized Frobenius traces. Through mechanistic interpretability techniques, they identified a sparse circuit of 20 MLP neurons sufficient for prediction, implementing a push-pull detector architecture. Notably, the model's learned input weights closely matched the Mestre-Nagao sum heuristic, indicating it learned a result from analytic number theory directly from the data. AI
IMPACT Demonstrates transformers' capability to learn complex mathematical heuristics, potentially opening new avenues for AI in theoretical sciences.