AI Mime：为计算机使用智能体构建高保真工作流的RPA工具

Core Idea: AI Mime is a native macOS RPA tool designed to solve the context shortage problem for computer-use agents. It uses a three-stage recording-refining-replaying workflow to convert human demonstrations into reliable, adaptable automation.

Basic Info:

• Author/Maintainer: prakhar1114
• Source: GitHub (https://github.com/prakhar1114/ai_mime)
• Version: v1.0.0
• Supported Platform: macOS

Key Value: Bridges the gap between lab demo computer-use agents and production-ready RPA by capturing rich context from human operations.

Traditional RPA tools rely on predefined scripts and fixed UI locators, which fail to adapt to dynamic modern interfaces. Computer-use agents (powered by VLMs) face a critical issue: natural language instructions often omit key details (e.g., edge cases, file save paths), leading to cumulative errors in long tasks. AI Mime addresses this context deficit.

1. Recording: Captures mouse/keyboard events, screenshots, and optional voice notes. Saves data to recordings/<session_id>/ (manifest.jsonl, screenshots, audio, metadata.json).

2. Refining: AI-driven process (VLM) transforms raw data into structured, parameterized workflows:

• Intent analysis
• Subtask decomposition
• Parameter extraction (e.g., contact name, message text)
• Dependency setup Example schema.json for WhatsApp message workflow included.

3. Replaying: Orchestrator manages subtask order/dependencies; inner loop (observe→infer→update memory→execute→check completion) uses VLM to adapt to UI changes.

VLM Role:

• Refine stage: Understand user intent from recordings.
• Replay stage: Visual reasoning to execute actions (supports OpenAI, Google Gemini, DashScope via LiteLLM).

Modular Design: Recording, refine (Reflect), replay, editor (browser-based), menu bar app.

Security: Requires macOS permissions: Accessibility (monitor input), Screen Recording (capture screenshots), Input Monitoring (keyboard events). Note: Add terminal and Python binary to permission lists.

Personal: Automate daily tasks (报销 forms, report generation, social media management, data entry).

Enterprise: Standardize processes, create training materials, ensure quality, preserve knowledge (when employees leave).

Dev/QA: UI test automation, regression testing, cross-platform testing (different macOS versions).

Current Limitations: macOS-only, network-dependent (VLM calls), performance overhead, limited error recovery.

Future Plans: Local model support (solve privacy/network issues), cross-platform expansion (Windows/Linux), smart error recovery, collaboration features, integration with existing RPA tools.

Paradigm Shift: From script writing to human demonstration (lower barrier for non-technical users).

Human-AI Collaboration: Humans demonstrate tasks; AI converts to reusable workflows and adapts to changes.

Context Engineering: Emphasizes providing rich context (beyond prompts) for reliable AI execution.