New framework uses soft prompts for multimodal emotion estimation

Researchers have developed a new multimodal framework for estimating valence-arousal (VA) in human emotions, utilizing Distance-aware Soft Prompt Guidance. This approach partitions the VA space into discrete regions, using Gaussian kernels to compute soft labels based on Euclidean distance, enabling finer-grained emotional transition learning. The framework integrates visual features from a CLIP image encoder and acoustic features from an Audio Spectrogram Transformer, with temporal modeling via Gated Recurrent Units and a hierarchical fusion scheme. AI

IMPACT Introduces a novel approach to emotion recognition by bridging semantic representations with continuous affective dimensions using soft prompts.

RANK_REASON The cluster contains a research paper detailing a new method for emotion estimation. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• Aff-Wild2
• Audio Spectrogram Transformer
• Byeongjin Jung
• CLIP image encoder
• Distance-aware Soft Prompt Guidance
• gated recurrent unit

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →