Open-Source LLMs Evolve: Attention, Multimodality, and Efficiency Gains

The open-source LLM landscape has seen significant shifts in recent months, with Sliding Window Attention becoming mainstream, enabling much larger context windows. QK-Norm is also gaining traction as a training stabilizer, tracing back to Gemini 3's architecture. Early multimodal pretraining, as seen in Kimi k2.5, is proving beneficial for reasoning, while GLM-5 from Z.ai, though modified, matches top proprietary models. Step 3.5 Flash stands out for its inference speed and multi-token prediction, though benchmark performance doesn't always align with user preference. AI

IMPACT New architectural innovations like Sliding Window Attention and QK-Norm are enabling more efficient and capable open-source LLMs, potentially lowering barriers to advanced AI development.

RANK_REASON The article discusses advancements in open-source LLM architectures and training techniques, including new attention mechanisms and pretraining strategies, rather than a specific model release from a frontier l [lever_c_demoted from research: ic=1 ai=1.0]

Read on dev.to — LLM tag →

• DeepSeek-V2
• Gemini 3
• GLM-5
• GPT-5.2
• Grouped-Query Attention
• Kimi k2.5
• MiniMax M2.5
• Opus 4.5
• QK-Norm
• Sliding Window Attention
• Step 3.5 Flash
• Z.ai

• model release
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →