ROMA system uses LLMs for active, multi-sensory perception

Researchers have developed ROMA, a new Large Language Model (LLM) system designed for active perception in real-world scenarios. Unlike traditional systems that passively integrate sensory data, ROMA actively seeks missing information by interacting with its environment. The system integrates vision, audio, tactile, and force sensing, and utilizes a reasoning-interaction-feedback loop to identify and acquire necessary evidence. To support this, a large-scale dataset called ROMI-2K, featuring nearly 2,000 objects and 6 atomic interactions, was created. ROMA has demonstrated its ability to solve complex, multi-sensory perception tasks that require long chains of reasoning and interaction. AI

IMPACT This research could enable more sophisticated embodied AI agents capable of complex real-world interaction and understanding.

RANK_REASON The item describes a new research paper detailing a novel system and dataset for active perception. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• arXiv
• Hugging Face
• LLM
• ROMI-2K

• paper
• product

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →