New YOLO-based system enhances robot gesture recognition for multimodal interaction

Researchers have developed a new cloud-edge multimodal interaction system for robots designed to improve human-robot interaction in environments with limited onboard computing power. The system integrates an enhanced YOLO-based gesture detector, which incorporates the Convolutional Block Attention Module and Distance-IoU loss for better gesture recognition and localization. This detector works in tandem with large language model (LLM) and vision-language model (VLM) agents. The cloud layer handles complex tasks like gesture detection and action planning, while the robot executes actions and provides feedback. Experiments show high precision and mAP values for gesture detection, with successful task completion rates of up to 95% for single-action tasks and a user satisfaction score of 3.69 out of 5. AI

IMPACT Enhances robot interaction capabilities by improving gesture recognition and task planning in resource-constrained environments.

RANK_REASON The cluster contains a research paper detailing a new system for robots.

Read on arXiv cs.AI →

• arXiv
• Convolutional Block Attention Module
• DagsHub
• Distance-IoU loss
• Hugging Face
• large language model
• TonyPi robot
• vision-language model
• YOLO
• YOLO-DC

• paper
• product
• infra

AI-generated summary · Google Gemini · from 2 sources. How we write summaries →