New AdaMTP paradigm improves LLM training with adaptive prediction

Researchers have introduced AdaMTP, an adaptive training paradigm designed to improve Multi-Token Prediction (MTP) for large language models. Unlike existing MTP frameworks that use a fixed prediction horizon, AdaMTP dynamically adjusts the prediction length based on the sequence's predictability, using an entropy-based segmentation algorithm to identify semantic boundaries. This approach aims to mitigate noisy training signals and gradient interference that can degrade the model's core capabilities. Experiments on Llama-3.1-8B, Qwen 2.5 7B, and Gemma-3-12B models demonstrated that AdaMTP enhances performance and inference speed across various benchmarks. AI

IMPACT AdaMTP could lead to more efficient and effective training of large language models, improving performance and inference speed.

RANK_REASON The cluster contains a research paper detailing a new training paradigm for LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• AdaMTP
• arXiv
• Gemma-3-12B
• Hugging Face
• large language model
• Llama-3.1:8b
• Multi Token Prediction
• Qwen 2.5 7B

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →