Next-Latent Prediction Transformers Learn Compact World Models
Researchers have developed a new training method called Next-Latent Prediction (NextLat) for transformers, which encourages them to build more compact internal world models. This approach adds a self-supervised objective to standard next-token prediction, training the transformer to predict its future latent state based on the current token. The method has shown empirical gains in accuracy, representation compression, and planning across various benchmarks, including language modeling where it also accelerates inference. AI
IMPACT Enhances transformer capabilities by enabling more efficient internal world models, potentially improving generalization and inference speed.