Regret Pre-training: Bridging Prior and Posterior Views for Enhanced Knowledge Grounding
Researchers have developed a new self-supervised learning framework called Regret Pre-training to improve causal language models. This method leverages future information typically unavailable during standard causal training by using a dual-view architecture. The framework trains a model to generate both a causal student distribution and a future-conditioned teacher distribution, minimizing the divergence between them to transfer future-aware signals. Experiments on nine downstream tasks showed significant accuracy improvements, with one configuration boosting BoolQ performance by over 18 percentage points. AI
IMPACT This framework could lead to more knowledgeable and accurate language models by effectively utilizing all available training data.