Brief

Researchers have introduced Nemotron 3 Ultra, a 550 billion parameter language model that utilizes a hybrid Mamba-Transformer architecture with a Mixture-of-Experts approach. The model was trained on 20 trillion tokens and features a 1 million token context length, along with advanced techniques like LatentMoE and Multi Token Prediction. Nemotron 3 Ultra demonstrates up to six times higher inference throughput than current state-of-the-art models while maintaining comparable accuracy, making it suitable for complex agentic tasks. The model's checkpoints, training data, and recipe have been open-sourced on Hugging Face. AI

Researchers have developed a new method called Counteraction-Aware Multi-Teacher On-Policy Distillation (CaMOPD) to address the challenge of recovering general capabilities in large language models (LLMs) after domain specialization. Existing methods often struggle when the training data distribution of general teachers is unknown. CaMOPD tackles this by using decoupled alternating training and a gap-based sample selection strategy. This approach allows for dedicated updates for general recovery, periodic checks for domain preservation, and focuses correction signals on samples with larger teacher-student log-probability gaps. Experiments show CaMOPD outperforms baselines in general recovery while maintaining domain-specific behavior in scenarios like role-play dialogue and medical reasoning. AI