Vision-language models struggle with fine-grained thermal activity recognition for robots

A new study published on arXiv investigates the performance of vision-language models (VLMs) when applied to thermal imagery for human activity understanding in human-robot interaction (HRI). The research found that while these models can identify general activity categories, they struggle with recognizing fine-grained actions like coughing or yawning, often hallucinate events, and produce inaccurate timelines. The study evaluated seven VLMs, including those designed for robotics, and found that simple preprocessing and additional reasoning provided only limited improvements, highlighting reliability concerns for practical robotic applications. AI

IMPACT Highlights limitations of current vision-language models for real-world robotic perception in challenging conditions, indicating a need for improved fine-grained action recognition and reliability.

RANK_REASON The cluster contains an academic paper detailing research findings on the failure modes of existing AI models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• arXiv
• embodied artificial intelligence
• Human Activity Understanding
• Human-Robot Interaction Using Affective Cues
• RGB cameras
• RGB color model
• Robots
• Thermal HRI
• vision-language model

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →