GUIDE Benchmark: How GUI Intelligent Assistants Move from Automation to True Collaboration

The GUIDE benchmark focuses on the collaborative capabilities of GUI intelligent assistants, revealing the shortcomings of current multimodal models in understanding users' operation intentions, while proving that providing structured user context can increase help prediction accuracy by 50 percentage points. This benchmark marks a paradigm shift in GUI agent research from "automation" to "true collaboration."

Traditional GUI agent research focuses on automation (doing operations on behalf of users), but ignores users' needs for exploration and iterative thinking. A truly intelligent assistant needs to understand users' behaviors and intentions and provide help at the right time—this is the core capability evaluated by the GUIDE benchmark.

GUIDE (GUI User Intent Detection Evaluation) is a benchmark for evaluating AI collaborative intelligence. The dataset includes 67.5 hours of screen recordings, operations from 120 novice users, 10 software applications, and synchronized voiceovers. Core tasks include: 1. Behavior state detection (identifying user states of exploration/difficulty/completion); 2. Intent prediction (inferring users' final goals); 3. Help prediction (determining the timing and method of assistance).

Testing 8 advanced multimodal models found: the average accuracy of behavior state detection is 44.6%, and help prediction is 55.0%—overall performance is unsatisfactory. However, when structured context (user skills, task goals, historical operations, etc.) is provided, the help prediction accuracy can be increased by up to 50.2 percentage points.

Difficulties in GUI understanding include: 1. Complex multimodal fusion (integration of visual, temporal, and semantic information); 2. Unpredictable open-ended tasks (users dynamically adjust their goals); 3. Difficulty balancing help timing (interrupting too early or helping too late).

The GUIDE results indicate the need to: 1. Shift from general to personalized (user profiling, historical memory, preference learning); 2. Balance active and passive assistance; 3. Enhance multimodal architectures (temporal modeling, visual attention, intent modules).

The capabilities evaluated by GUIDE can be applied to: 1. Built-in software intelligent assistants; 2. Accessibility assistive technologies; 3. Remote collaboration and training; 4. Automated testing and quality assurance.

GUIDE marks the shift of GUI agent research towards collaborative intelligence, with the core being understanding users rather than replacing their operations. Its limitations include a bias towards novice users, limited software coverage, and real-time performance that needs optimization. In the future, we need to expand user groups and software types, solve real-time response issues, and promote a new era of human-computer collaboration.