HippoCamp: A New Benchmark for Evaluating Context-Aware Agents on Personal Computers

HippoCamp is a new multimodal file management agent evaluation benchmark. Built using 42.4GB of real user data to create 581 question-answer pairs, it reveals that state-of-the-art models only achieve an accuracy of 48.3% in user profile modeling and cross-modal reasoning, highlighting their performance bottlenecks. This benchmark focuses on evaluating the capabilities of context-aware agents in personal computer environments, providing a rigorous testing platform for the development of personal AI assistants.

Current large language models and agents are primarily developed for scenarios like web interaction and tool calling. However, practical personal AI assistants need to handle massive private files in personal computer environments, understand personalized needs, and perform context-aware reasoning. Existing evaluation benchmarks are detached from real scenarios (controlled experiments or single modalities), leading to excellent lab models performing poorly in real personal file systems. Users need assistants that "understand" them (remember preferences, locate documents, cross-modal reasoning).

Design Philosophy: Named after the hippocampus (responsible for memory and navigation), the core goal is to evaluate agents' memory, retrieval, and reasoning abilities in personal digital environments. It adopts a user-centric design and processes messy multimodal data based on real user profiles. Dataset Composition: Contains 42.4GB of real data (2000+ files across text documents, images, etc., multimodal), 581 deep reasoning question-answer pairs, and 46,100 densely annotated structured trajectories (supporting fine-grained failure diagnosis). Evaluation Dimensions: Search ability (semantic retrieval, intent understanding), evidence awareness (multimodal content understanding and relevance assessment), multi-step reasoning ability (task decomposition, plan adjustment, metacognition).

Evaluations of current state-of-the-art multimodal models and agents show that the best commercial models only achieve an accuracy of 48.3% in user profile modeling tasks. Key bottlenecks:

1. Long-range retrieval: Prone to getting lost when searching across months/folders, premature convergence or resource waste, reflecting limitations in long-context processing;
2. Cross-modal reasoning: Performance drops significantly when integrating evidence from different modalities (e.g., email text + attached images), and multimodal fusion remains unsolved.

Through structured trajectory analysis, two major performance bottlenecks are identified:

1. Multimodal perception issues: Weak ability to understand non-text content (charts, image scenes, audio-visual information), making it hard to connect to task goals;
2. Evidence grounding issues: Over-reliance on insufficient evidence or misuse of correct evidence, failing to effectively link information to the reasoning process.

For researchers: Provides a rigorous testing platform to guide the identification of technical limitations and research directions; For developers: Need to strengthen memory systems (efficiently organize long-term information), cross-modal understanding (core skill), interpretability, and debuggability; For users: Current personal AI assistants still have a gap in "understanding" users. Caution is needed in privacy-related scenarios, and it's important to understand technical limitations. HippoCamp marks a new stage in personal AI assistant evaluation, directly addressing real-world complexity and helping develop useful and reliable personal AI assistants.