VOCO: An Architecture Analysis of a Fully Local-Running Autonomous AI Agent System

VOCO is a fully local offline autonomous AI agent system based on the Ollama local large model runtime, supporting browser automation, desktop operations, file management, code generation, and other capabilities. Its core value lies in addressing privacy and security concerns of cloud-based AI agents and usage issues in network-constrained scenarios. It adopts a hybrid routing architecture design to provide feasible solutions for privacy-sensitive and network-constrained scenarios.

Project Background

Most AI agents rely on cloud APIs, which pose risks of privacy data leakage and cannot be used in network-constrained environments. VOCO was developed to provide a locally offline-running alternative.

Core Features

• Fully local operation: Based on Ollama, all data processing is completed on the user's machine with no third-party server dependencies.
• Applicable scenarios: Privacy-sensitive environments (commercial secrets/personal privacy processing), network-constrained scenarios (airplanes/remote areas/enterprise intranets), cost control (no API call fees), customization needs (deeply adapting to personal workflows).

System Capabilities Overview

• Browser automation: Control browsers to perform navigation, input, clicks, etc., and convert natural language instructions into operation sequences.
• Desktop application control: Execute multi-step workflows across software (e.g., Excel data processing → PowerPoint report generation).
• File and index search: Semantic retrieval of local file content.
• Dedicated processes: Deterministic workflows like YouTube comment export, code generation and repair, report generation.
• Voice interaction: Local speech recognition model supports one-click voice command input.

Hybrid Routing Architecture

• Deterministic fast path: Predefined rules handle clear intents (e.g., opening apps, file operations) with low latency.
• Routing family contracts and classifier guardrails: Group by task type; classifiers judge intent归属 and evaluate confidence to prevent blind operations.
• Tool-first decomposition: Decompose complex tasks into tool calls to improve reliability.
• LLM fallback mechanism: Call LLM reasoning only when necessary to optimize resource efficiency.

System Layered Design

UI layer (terminal dashboard), orchestrator layer (task planning and coordination), router layer (intent recognition and routing), tool layer (specific function implementation), memory layer (context and history management), evaluation layer (testing and benchmarking).

Deployment Environment Requirements

• Windows 10/11 operating system
• Python 3.10+ version
• Ollama installed and PATH configured
• Playwright browser dependencies (auto-installed on first run)

Configuration Options

• Default uses qwen3:4b model; modify constants.py to adjust model, context length, autonomy mode, and other parameters.
• Autonomy control options: AUTONOMY_MODE (autonomy level), HUMAN_APPROVAL_DISABLED (manual confirmation switch).

Quality Assurance

• Reliability test suite: python eval.py suite verifies core function stability.
• Benchmark test: python eval.py benchmark evaluates task performance.
• Misroute guardrail test: python eval.py benchmark --category misroute --no-gate detects classifier accuracy.
• Decomposition regression test: python test_decomp.py verifies task decomposition logic.

Applicable Users and Scenarios

• Privacy-first users: Professionals handling sensitive data such as lawyers, doctors, and financial practitioners.
• Automation enthusiasts: Technical users who want to automate repetitive operations.
• Offline workers: Business people who travel frequently and work in network-free environments.
• AI developers: Reference case for learning local agent system construction.

Usage Recommendations

Adjust the autonomy mode and model configuration according to needs; it is recommended to enable the manual confirmation function in production environments to ensure operational safety.

Limitations

• Model capability boundary: Local model reasoning ability is weaker than top cloud models (e.g., GPT-4).
• Hardware requirements: Although the 4B model has a low threshold, a better experience requires certain computing resources.
• Ecosystem: The local toolchain and integration ecosystem are still in development.

Future Outlook

With the progress of open-source models and the improvement of edge computing capabilities, the limitations of local AI agents will gradually narrow. VOCO demonstrates a feasible path for local distributed deployment, indicating the trend of fully local-running personal AI assistants.