LLM-DOM-Agent: Autonomous Browser Automation Agent Based on Large Language Models

LLM-DOM-Agent is an open-source browser automation tool that combines browser extensions with a local Python server to achieve autonomous web browsing and information extraction using large language models (LLMs). It addresses the pain point of traditional automation tools relying on predefined selectors and struggling to adapt to dynamic web pages. Adopting a dual-component architecture and a perception-reasoning-action loop workflow, it supports natural language instruction-driven operations, has adaptive fault tolerance capabilities, and has broad application potential in multiple scenarios such as automated testing and data scraping.

Browser automation is a classic problem in software engineering. Traditional tools like Selenium and Playwright rely on predefined DOM selectors/XPath and struggle with dynamic web pages. With the rise of LLMs, their ability to understand natural language, reason about page content, and generate operation code has spawned a new automation paradigm. LLM-DOM-Agent is a representative work of this trend, combining LLM reasoning capabilities with browser automation to achieve more intelligent and adaptive web interactions.

Dual-Component Architecture:

• Browser extension: Responsible for DOM information extraction, element marking, action execution, and status feedback;
• Local Python server: Acts as the 'brain', handling LLM interactions, decision engine, task management, and error handling.

Workflow follows the perception-reasoning-action loop:

1. Perception: Extract DOM structure, interactive elements, and other information, then generate an intelligent summary;
2. Reasoning: The LLM receives task instructions and DOM information, understands the intent, analyzes the state, and plans actions;
3. Action: The browser extension executes the operations decided by the LLM (clicking, inputting, etc.);
4. Loop: Repeat the above steps until the task is completed or limits are reached.

Natural Language-Driven: Users only need to describe the goal in natural language (e.g., 'Find the price of iPhone 16') without writing scripts; Adaptive and Fault Tolerant: Adapts to page changes, handles dynamic content, error recovery, and multi-path exploration; Privacy and Security: Processes DOM information locally, user controls operations, no persistent storage, open-source and transparent code.

LLM-DOM-Agent can be applied in:

• Automated testing: Write test cases in natural language;
• Data scraping and monitoring: Extract data from dynamic websites and monitor price changes;
• Assisted browsing: Navigate web pages via voice commands;
• Automatic form filling: Intelligently identify fields and fill them;
• Workflow automation: Execute business processes across web applications (e.g., download attachments → upload to cloud storage → create tasks).

Current limitations include:

• Cost: High LLM API call fees;
• Latency: Time-consuming network round trips and LLM reasoning;
• Accuracy: LLMs may misinterpret page content or make wrong decisions;
• Security: Automatic operations have risks and require permission control;
• Context limitations: Complex DOM may exceed the LLM's context window.

Future development directions include:

• Multimodal capabilities: Combine visual models to recognize page screenshots;
• Learning optimization: Build an operation pattern library to reduce LLM dependency;
• Local model support: Integrate lightweight local LLMs to reduce costs;
• Cross-platform expansion: Support desktop/mobile automation;
• Collaboration features: Multi-agent collaboration to handle complex tasks.

LLM-DOM-Agent demonstrates the potential of integrating AI with browser automation, pioneering a more natural and intelligent human-computer interaction method. Although it is in the early stage, its design concept provides a reference for intelligent agent systems. As LLM capabilities improve and costs decrease, such tools will play an important role in fields like automated testing and data scraping, and also provide developers with an excellent case of integrating LLMs with existing systems.