Training-Trajectory-Aware Token Selection
Researchers have developed a new method called Training-Trajectory-Aware Token Selection (T3S) to improve the efficiency of distilling knowledge from large language models. This technique addresses a common issue where performance metrics can drop during distillation, even as the loss decreases. T3S works by reconstructing the training objective at the token level, which helps clear the optimization path for tokens that are still learning. The method has shown consistent gains in various settings, with T3S-trained models achieving state-of-the-art performance among models of similar scale. AI
IMPACT Improves efficiency in distilling large language models, potentially leading to more capable and accessible models.