HippoCamp: The Capability Boundaries of AI Agents When Facing Real Personal Computer File Systems

The research team from Nanyang Technological University released the HippoCamp benchmark, which for the first time systematically evaluates the real-world performance of multimodal large models in personal computer file management scenarios. Based on 42.4GB of real user data, the test reveals that the current state-of-the-art commercial models only achieve an accuracy rate of 48.3% in user profiling tasks, with multimodal perception and evidence localization being the main capability bottlenecks. The launch of HippoCamp provides the industry with a standardized evaluation tool for real-world scenarios, helping to explore the capability boundaries of AI agents.

Existing AI agents perform well in general scenarios such as web browsing and tool calling, but there is a fundamental gap when applied to real personal computing environments—can they understand and manage the massive personal files scattered across users' computers? Traditional benchmarks mostly focus on general scenarios and ignore user-centric real file scenarios (e.g., massive file search, user habit understanding, cross-modal reasoning), so the NTU team carried out the HippoCamp research.

HippoCamp focuses on user-centric environment evaluation, with its core innovation lying in the real test environment: a 42.4GB dataset built based on real user profiles, containing over 2000 multimodal files (text documents, images, audio, etc.); test dimensions cover search capabilities (deep intent understanding), evidence perception (multimodal information integration), and multi-step reasoning (cross-file association); it provides over 46,100 densely annotated trajectories for fine-grained failure analysis.

Evaluation results show that the state-of-the-art commercial models only achieve an accuracy rate of 48.3% in user profiling tasks. Major challenges include long-range retrieval (tracking information in massive files is easily disturbed) and cross-modal reasoning (insufficient ability to associate information from different modalities); key shortcomings are multimodal perception (limited understanding of image details) and evidence localization (finding files but making mistakes in extracting explanatory content).

Three capabilities need to be improved: 1. Multimodal fusion (stronger cross-modal attention mechanisms to enable free association of text, visual, and structured data); 2. Long context processing (focus on extremely long contexts and flexible jumps); 3. Personalized understanding (adapting to users' file organization habits, preferences, and knowledge structures).

HippoCamp fills the gap in standardized evaluation of real personal environments, prompting developers to focus on real-world deployment performance; it provides an objective evaluation benchmark for personal AI assistant products; it reminds us to be cautious when applying AI in sensitive personal data environments—if agents cannot accurately understand file content, granting high decision-making authority may bring risks.