Brief

Researchers have explored alternative objectives for supervised fine-tuning (SFT) of large language models, moving beyond the standard negative log likelihood (NLL). Their study, involving extensive experiments across various models and benchmarks, reveals that different objectives perform better depending on the model's capability. Objectives that downweight low-probability tokens are more effective for highly capable models, while NLL excels with less capable models. AI

Anyscale has introduced a new Anyscale Agent Skill designed to simplify and automate the process of generating LLM post-training runs. This skill assists users in selecting the most appropriate post-training method, such as SFT, CPT, DPO, or RLVR, based on their model, dataset, and objectives. It then generates configuration files for popular frameworks like LLaMA-Factory and Ray Train, preparing them for deployment on Anyscale Jobs. AI

Researchers have found that Supervised Fine-Tuning (SFT) using outputs from a different AI model can significantly degrade the capabilities of the trained model. This degradation appears to be linked to the model adopting an unfamiliar reasoning style that it struggles to utilize effectively. The issue is not necessarily due to imitating a less capable teacher model, as degradation occurs even when the teacher is superior. Fortunately, this performance drop seems to be a shallow property, as a small amount of training to restore the original reasoning style can recover most of the lost performance. AI