AutoGLM: An Android Phone Intelligent Agent Framework Based on Vision-Language Models

AutoGLM is an open-source Phone Agent framework that can automatically control Android phones via natural language instructions. It combines vision-language models, ADB/HDC debugging tools, and multimodal understanding capabilities, supporting over 50 mainstream Chinese apps and providing a complete solution for mobile automation and intelligent assistant development. The project is maintained by GitHub user badhope, based on the official open-source project Open-AutoGLM from Zhipu AI, created in February 2026 and continuously updated. Original link: https://github.com/badhope/AutoGLM.

With the development of large language models and vision-language models, AI Agents are evolving toward multimodality and cross-platform capabilities. Mobile users perform many daily operations, but repetitive tasks still require manual completion. Traditional tools like Appium require complex scripting, which has a high barrier to entry. AutoGLM combines natural language understanding with visual perception—users only need to describe their needs in one sentence, and the AI can understand the screen, plan steps, and execute them, achieving a "what you see is what you get" interaction. This is a significant breakthrough in mobile Agent technology.

Core technologies include: 1. Vision-language model-driven: The AutoGLM-Phone-9B model (based on the GLM-4.1V-9B-Thinking architecture), optimized for Chinese and supporting multilingual versions, outputs structured operation instructions via chain-of-thought; 2. ADB/HDC action execution: Supports various operations like Launch, Tap, Type, and requests manual takeover in sensitive scenarios; 3. Multi-platform model support: Built-in presets for 18 AI service providers, allowing one-click switching between Zhipu BigModel, OpenAI, Google Gemini, etc.

Application scenarios cover over 50 mainstream Chinese Android apps (social communication, e-commerce shopping, life services, etc.). Typical scenarios include cross-platform price comparison, automated schedule management, social message processing, travel planning, content retrieval, etc. In terms of security and privacy design, sensitive operations (payment, password input) are automatically detected and request manual takeover, supporting custom confirmation callbacks and detailed log output.

Deployment requires preparing an Android 7.0+ device (with Developer Mode and USB Debugging enabled), ADB tools, ADB Keyboard, and Python 3.10+. Model services can be selected as cloud APIs (recommended for beginners, such as Zhipu BigModel) or local deployment (requiring 24GB+ VRAM). Supports WiFi remote ADB/HDC debugging without USB connection.

Technical highlights include: 1. End-to-end visual understanding: Directly understands interfaces via screenshots with strong generalization ability; 2. Explicit chain-of-thought: The model's reasoning process is visible, enhancing interpretability; 3. Multimodal action space: Unified operation output facilitates learning and execution; 4. Platform independence: Supports Android and HarmonyOS; 5. Open ecosystem integration: Adapts to SDKs like Midscene.js, supporting JS/YAML to define complex processes.

Limitations include: 1. Device compatibility: Customized systems may have issues, requiring specific debugging settings to be enabled; 2. Sensitive page restrictions: Black screen protection in apps like bank payment requires manual takeover; 3. Network dependency: Cloud models need a stable network; 4. Learning curve: Initial deployment requires environment and dependency configuration.

AutoGLM represents a significant advancement in the mobile AI Agent field. It combines large language model reasoning with computer vision perception, shifting mobile automation from "script-driven" to "intent-driven". For developers, it is a valuable project for researching multimodal Agents; for users, it demonstrates the potential of future human-computer interaction. With the improvement of on-device models and device computing power in the future, it is expected to play a greater role in intelligent assistants, accessibility support, and other fields, and its open-source ecosystem is worth paying attention to.