Temperon training method achieves SAM quality with reduced cost

Researchers have introduced Temperon, a novel training method designed to achieve the quality of Sharpness-Aware Minimization (SAM) while significantly reducing computational costs. Temperon utilizes a two-phase approach: an initial exploration phase with standard SGD followed by a hand-off to a SAM-wrapped Muon refiner for the latter part of the training. This method has demonstrated comparable accuracy to full-time SAM on various datasets like CIFAR-10/100 and Tiny ImageNet, achieving target accuracies faster. The approach has also shown effectiveness in pretraining GPT-2 and fine-tuning for GLUE tasks, offering substantial wall-clock time savings. AI

IMPACT This method could significantly reduce the computational resources required for training large models, making advanced techniques more accessible.

RANK_REASON The cluster contains a research paper detailing a new training methodology for machine learning models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• CIFAR-10
• CIFAR-100
• Glue
• GPT-2
• muon
• Sam
• SGD
• Temperon
• The Street View House Numbers Dataset
• Tiny-ImageNet

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →