DragOn: A Benchmark and Dataset for Drag-Based GUI Interactions
Researchers have introduced DragOn, a new benchmark and dataset designed to improve the performance of GUI agents on drag-based interactions. The dataset includes 286,000 training screenshots and 3.5 million training tasks across four domains: text highlighting, cell selection, element resizing, and slider manipulation. Evaluations on proprietary and open-weight models, including a fine-tuned Qwen VLM, suggest that DragOn can enhance the capabilities of state-of-the-art models for complex computer-use tasks. AI
IMPACT Enhances AI's ability to automate complex GUI interactions, potentially accelerating digital task automation.