New SAGE framework unifies action and gaze recognition and anticipation

Researchers have introduced SAGE, a novel framework designed to simultaneously recognize and anticipate human actions and gaze patterns. Unlike previous models that treated these elements separately, SAGE integrates them into a single, end-to-end trainable model. This unified approach utilizes a transformer-based architecture with spatiotemporal attention mechanisms that incorporate gaze data to predict both current and future actions and gaze behavior. To support further research, a new benchmark dataset called Exo-Cook has also been established. AI

IMPACT This research could lead to more intuitive human-machine interactions by better understanding human intent through synchronized action and gaze analysis.

RANK_REASON This is a research paper introducing a new model and dataset. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• EGTEA Gaze+
• Exo-Cook
• SAGE
• spatiotemporal attention mechanisms
• transformer-based architecture
• VidHOI

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →