Researchers pinpoint "first-token broadcasters" controlling language identity in transformers

Researchers have identified specific attention heads in transformer models, termed "first-token broadcasters," that are crucial for maintaining a model's language identity. These heads, particularly prominent in instruction-tuned models like Qwen2.5-1.5B-Instruct, persistently attend to the initial token of a prompt, thereby propagating its language signal throughout the generation process. Experiments with GPT-2 and Qwen2.5 models revealed that instruction tuning significantly reorganizes these language identity circuits, localizing their influence to earlier layers, while base models show a more distributed pattern. AI

IMPACT Provides a mechanistic explanation for language switching in LLMs, potentially guiding future model development for improved multilingual capabilities.

RANK_REASON Academic paper detailing a new mechanistic understanding of transformer model behavior. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• Arjun Pillai
• GPT-2
• Hugging Face
• Language Identity Head Ablation
• Qwen2.5-1.5B-Base
• Qwen2.5-1.5B-Instruct

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →