Learning What to Predict: Downstream-Guided Task Design for Continued Pretraining
Researchers have developed a novel method called V-pretraining to enhance the effectiveness of continued pretraining for AI models. This technique uses a small set of downstream examples to provide step-level feedback, guiding the model's learning process without directly supervising it with labels. V-pretraining has demonstrated improvements in specific target capabilities across both language and vision modalities, notably achieving a significant gain on the GSM8K benchmark for Qwen models. AI
IMPACT Introduces a method to improve target capabilities in AI models during pretraining without degrading general performance.