Multimodal AI models show shared mechanisms for processing speech and facial emotions

Researchers have investigated how multimodal foundation models (MFMs) process emotions from speech and facial expressions. By examining specific neurons within models like Gemma-4-12B-it, MiniCPM-o-4.5, and Qwen2.5-Omni-7B, they identified emotion-sensitive neurons (ESNs) that are crucial for recognizing affective cues. The study found that these ESNs play a causal role in emotion recognition, and their activations can be manipulated to enhance or impair the recognition of specific emotions. Furthermore, the research suggests a partial overlap and structural alignment in how these models represent emotions across both auditory and visual modalities, indicating shared affective mechanisms. AI

IMPACT Reveals insights into how AI models process and represent emotions, potentially guiding future multimodal AI development.

RANK_REASON The cluster contains an academic paper detailing research findings on multimodal foundation models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• alphaXiv
• arXiv
• CatalyzeX
• DagsHub
• Gemma-4-12B-it
• Gotit.pub
• Hugging Face
• Qwen2.5-Omni-7B
• ScienceCast

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →