Supervised Fine-tuning with Synthetic Rationale Data Hurts Real-World Disease Prediction
A new research paper challenges the common assumption that supervised fine-tuning with synthetic rationale data improves language model performance on clinical prediction tasks. Experiments on Alzheimer's disease prediction found that this method consistently degraded performance compared to label-only fine-tuning, even when the rationales were medically accurate. The study suggests a conflict between narrative plausibility and discriminative optimization is the likely cause, urging caution in developing language models for high-stakes medical applications. AI
IMPACT Challenges the efficacy of rationale-based fine-tuning for high-stakes clinical prediction, suggesting a need for more robust training methodologies.