Researchers have explored how controlling the routing of tokens in mixture-of-experts (MoE) language models can influence their output. By biasing tokens towards specific expert paths, they found that format-related steering significantly alters the generated text, while concept-related steering has a lesser impact on the final output despite shifting next-token probabilities. This effect is concentrated in the middle layers of the model, with concepts like religion and money showing the strongest influence. AI
IMPACT Provides insights into controlling and understanding the internal workings of MoE models, potentially leading to more steerable and predictable AI systems.
RANK_REASON Academic paper detailing novel research into MoE model behavior. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →