New decoder architecture shows improved retention in small code models

Researchers have developed SamatNext v0.2-B, a 356M-parameter hybrid sequence decoder designed to mitigate forgetting in small code models during curriculum learning. This experimental model alternates Differential-Attention-style layers with simplified linear-state mixer layers, employing RMS normalization and output scale calibration. In controlled tests on a Python code curriculum, SamatNext v0.2-B achieved a 100.0% pass rate on a later stage while retaining 98.8% of earlier stage semantic behavior, significantly outperforming a parameter-matched Transformer baseline in retention. AI

IMPACT Introduces a novel decoder architecture that may improve curriculum retention and reduce forgetting in small code models.

RANK_REASON This is a research paper detailing an experimental model architecture and its performance on specific benchmarks. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• SamatNext v0.2-B
• Samat Zharassov
• Transformer

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →