VLM-Agent: A Visual Automation Framework Based on Vision-Language Models, with Go Client and Python Inference Service

VLM-Agent is a visual automation framework combining visual language models (VLM) and large language models (LLM). It adopts a gRPC architecture with a Go client and Python inference server, offering a new solution to GUI automation challenges faced by traditional methods. This framework allows AI to "see" screens like humans, understand interfaces, and execute operations, breaking free from reliance on underlying interface structures.

Traditional GUI automation tools have evolved from OS API calls to DOM-based (e.g., Selenium) and accessibility tag-based (e.g., Appium) methods. However, these rely on machine-readable interface structures, failing in scenarios like custom-rendered game interfaces, Canvas/WebGL visualizations, non-standard cross-platform UI, or DOM-obfuscated apps. VLM-Agent's core innovation is using VLM to eliminate this dependency.

VLM-Agent uses VLM (e.g., GPT-4V, Claude3, Qwen-VL) for "perception"—analyzing screen state, identifying interactive elements, and understanding layout. LLM handles "decision-making"—planning action sequences from task goals. This separation leverages each model's strengths and provides clear debugging/optimization interfaces.

The client uses Go for its small binary size, fast startup, low resource usage (ideal for background agents), and strong concurrency (supporting multi-window/task scenarios). The inference server uses Python (rich AI/ML ecosystem) and communicates via gRPC (efficient binary serialization, strong typing, better for high-frequency/low-latency calls). This separation combines Go's efficiency and Python's AI capabilities.

VLM-Agent excels in: 1) Complex interfaces (game-engine enterprise apps, legacy non-standard UI, custom SaaS products); 2) Cross-platform automation (works on Windows/macOS/Linux/mobile via screenshots); 3) Intelligent test automation (natural language test intent, reducing script maintenance and improving UI change robustness).

Key challenges include: 1) Latency (screen capture → VLM analysis → LLM decision takes seconds); 2) Cost (higher VLM API fees than text models); 3) Accuracy (errors in specialized interfaces, small text, complex tables); 4) Dynamic interfaces (current focus on static screenshots, needs frame extraction for video streams).

• vs RPA: No pre-recorded sequences or interface mapping, more adaptive; - vs Computer Vision+OCR: Understands semantic functions (e.g., "submit button" vs "blue area"); - vs Anthropic's Computer Use/OpenAI's Operator: Open-source, higher customizability and transparency for developers.

VLM-Agent represents a forward-looking direction—integrating multi-modal AI into automation. As VLM capabilities improve and costs drop, visual-based automation will become more practical. Future tools may interact with software via observation/understanding like humans. For AI application, test automation, or RPA professionals, VLM-Agent provides a valuable reference implementation, showing current feasibility and laying groundwork for innovation.