Research on Autonomous Testing System Based on Multimodal Computer Usage Model

This research focuses on using multimodal computer usage models to implement autonomous testing of software interfaces. By visually understanding GUI elements and simulating human operations, it aims to break through the bottlenecks of traditional automated testing (such as high maintenance costs and poor adaptability) and provide a new technical direction for the field of automated testing.

Software testing has evolved from manual to automated, but mainstream frameworks (Selenium, Appium, Playwright, etc.) have inherent limitations:

1. Fragility: Dependent on interface element positioning identifiers (ID, XPath, etc.), interface adjustments easily cause failures, and maintenance costs exceed development costs;
2. Limited expressive ability: Only executes predefined operations, making it difficult to handle abnormal scenarios (such as pop-ups, loading delays);
3. Limited test coverage: Difficult to design complex business scenarios, and the automation level of exploratory testing is low.

Multimodal large models such as GPT-4V and Claude 3 have visual + language fusion capabilities, opening up new paths for test automation:

• Robustness: No need for positioning scripts; directly perceives interface elements visually;
• Natural language conversion: Converts high-level instructions (e.g., "Test the user login function") into specific operation sequences;
• Reasoning and decision-making: Handles abnormal situations during testing;
• Close to the working mode of human testers, with human-like testing capabilities.

Building the system requires addressing multi-level challenges:

• Perception layer: Accurately identify interactive elements (buttons, input boxes, etc.) in screenshots, and understand their types, states, and semantics;
• Decision layer: Decompose test objectives into operational steps (e.g., step planning for shopping cart functions);
• Execution layer: Convert high-level instructions into low-level operations (e.g., mouse click coordinates);
• Verification layer: Determine interface state changes and the correctness of business logic (e.g., shopping cart amount calculation).

The system has application value in multiple scenarios:

• Regression testing: Automatically traverses core functions and adapts to interface adjustments;
• Cross-platform testing: The same instruction adapts to Web, iOS, Android, and other platforms;
• Exploratory testing: Independently explores operation paths and discovers boundary cases;
• Small and medium-sized enterprises/developers: Lowers the testing threshold—executes tests by describing intentions in natural language.

Current limitations:

• Visual understanding: Complex interfaces, small-sized elements, and non-standard controls are prone to misjudgment;
• Temporal uncertainty: Dynamic loading and asynchronous updates affect state perception;
• Cost and latency: High API call costs may affect CI/CD applications. Future directions:
• Optimize UI visual models and reduce dependence on general models;
• Fine-tuning/retrieval enhancement technologies to improve domain adaptability;
• Design a hybrid testing mode of human-machine collaboration.

This research represents the cutting-edge direction of software testing. Multimodal large models provide a new path to break through the bottlenecks of traditional testing, and the vision of "testing software like humans" is gradually becoming a reality. Although the technology is not yet mature, with the improvement of model capabilities and engineering refinement, autonomous testing systems are expected to become an important part of quality assurance. The industry will usher in a new mode of human-machine collaboration, where test engineers will shift to high-value activities such as strategy design, intention expression, and result analysis, promoting the development of testing toward efficiency and intelligence.