Qwen3-0.6B-Base model suffers from "interface injury" in attention linearization

Researchers have identified a specific issue in the linearization of attention layers in the Qwen3-0.6B-Base language model, where the model becomes overly reliant on answer labels rather than content. Despite achieving close perplexity to the original model through distillation, the linearized version performed poorly on multiple-choice tasks, consistently favoring the first option. A targeted KL distillation stage successfully repaired this "interface injury," significantly improving benchmark accuracy and reducing label-stickiness. AI

IMPACT Highlights a subtle but critical failure mode in model linearization that standard metrics miss, potentially impacting deployment of efficient models.

RANK_REASON Academic paper detailing a specific technical issue and its resolution in a language model. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• arXiv
• C Eval Benchmark
• Hugging Face
• Qwen3-0.6B-Base

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →