New framework unifies segmentation and VQA for robotic surgery

Researchers have developed a novel framework that unifies pixel-level segmentation and visual question answering (VQA) for robotic surgery. This approach uses object tokens generated by a vision-language model (VLM) to guide answer prediction and produce segmentation masks via a SAM-based decoder. By optimizing these object tokens with both segmentation and VQA objectives, the model learns spatially grounded representations that enhance reasoning and provide explicit pixel-level grounding. The method demonstrated superior performance on the RAMIE and EndoVis18 datasets, improving fine-grained surgical scene understanding. AI

IMPACT Enhances fine-grained surgical scene understanding and reasoning for robotic surgery applications.

RANK_REASON The cluster contains an academic paper detailing a new technical approach in computer vision. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• Computer vision and pattern recognition
• EndoVis18
• Object Tokens
• RAMIE
• robot-assisted surgery
• SAM
• vision-language model
• visual question answering

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →