New benchmark reveals safety reasoning gap in vision-language models

A new benchmark called SafeGesture has been developed to evaluate the fine-grained hand gesture understanding of vision-language models (VLMs) in safety-critical scenarios. The benchmark pairs six gestures with eight operational contexts, resulting in 4,800 evaluation items. Initial tests on models like GPT-4o and Qwen2.5-VL-7B revealed a significant gap between gesture recognition accuracy and safety action inference, indicating that reasoning about safety contexts is the primary challenge for current VLMs. AI

IMPACT Highlights a critical gap in VLM safety reasoning, suggesting future development should focus on contextual understanding rather than just gesture recognition.

RANK_REASON The cluster describes a new academic paper introducing a benchmark for evaluating AI models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• arXiv
• GPT-4o
• InternVL2-8B
• LLaVA-NeXT-7B
• Phi-3.5-Vision
• Qwen2.5-VL-7B
• SafeGesture

• paper
• safety
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →