Memorize Theorems, Not Instances: Probing SFT Generalization through Mathematical Reasoning
Researchers have developed a new method called Theorem-SFT to improve the generalization capabilities of supervised fine-tuned models. This approach shifts the focus from memorizing specific problem-solution pairs to understanding and applying explicit theorems. Theorem-SFT has shown significant performance improvements on mathematical reasoning benchmarks, including notable gains on MATH and GeoQA datasets when applied to LLaMA3.2-3B-Instruct and Qwen2.5-VL-7B-Instruct models. AI
IMPACT Enhances model reasoning by focusing on theorem application, potentially improving performance on complex tasks.