Optimizing Combinatorial Testing with Large Language Models: Intelligent Identification of Key Parameter Interactions

Project Name: Coverage_strength Original Author: mahdi943 Source: GitHub (link: https://github.com/mahdi943/Coverage_strength) Core Viewpoint: This project innovatively combines the natural language understanding and reasoning capabilities of large language models (LLMs) with traditional combinatorial testing techniques to intelligently identify high-risk parameter interaction combinations, implement a selective testing strategy, and aim to improve test coverage, reduce testing costs, and optimize test resource allocation.

Traditional combinatorial testing uses a t-way coverage strategy but has a "one-size-fits-all" limitation: treating all parameter combinations equally leads to low-risk combinations occupying large amounts of resources while high-risk combinations may receive insufficient attention. Project Motivation: Analyze software documents via LLMs to identify key parameter interactions, implement "selective" combinatorial testing, and solve the resource waste problem.

Core Components of the Technical Solution

1. LLM-driven Parameter Analysis: By understanding semantic information from API documents, Wikis, etc., infer parameter dependencies and conflicts, and identify high-risk combinations.
2. Coverage Array Generation: Use the Jenny tool to generate 2-way/3-way/4-way coverage arrays, minimizing the number of configurations.
3. Selective Testing Strategy:
   • document_only mode: Generate coverage arrays based solely on API documents
   • document_plus_wiki mode: Combine API documents and Wikis to generate more comprehensive coverage arrays

Supported Test Targets

For Apache open-source projects: Cassandra, Flink, Spark, HBase, JSPWiki, and other enterprise-level software.

System Requirements

Dependencies: Python 3.10+, Java JDK11+, Maven 3.6.3+, Ant 1.10+, Git 2.x, GCC, and other tools.

Code Coverage Collection

Use the JaCoCo tool, integrated via the Maven plugin, to collect coverage without modifying the POM of the tested project.

Test Execution Flow

1. Check out code → 2. Generate test configurations →3. Apply configurations →4. Build the project →5. Run tests and collect coverage →6. Archive results.

Expected Improvement in Test Efficiency

Take the Flink project as an example:

• document_only mode requires only 25 configurations
• document_plus_wiki mode requires only 31 configurations Compared to traditional 4-way coverage (hundreds to thousands of configurations), the number of test cases is significantly reduced.

Value of LLM-assisted Decision Making

If LLMs can effectively extract semantic information from documents, it will open up a new direction for software testing: using natural language processing to enhance traditional structural testing methods.

1. Intelligent Test Design: Shift from relying on manual experience/static analysis to test design based on semantic understanding, discovering more defects missed by traditional methods.
2. Cost-effectiveness Optimization: Intelligently allocate test resources to maximize test effectiveness within a limited budget.
3. Empowering Open-source Ecosystem: Provide efficient testing methods for widely used open-source projects like Apache, improving their quality and benefiting the dependent ecosystem.

Current Limitations

• Dependence on document quality: The effect of LLM analysis is affected by the completeness and accuracy of documents.
• Scope of application: Mainly supports Java/Maven projects; limited support for other languages/build systems.
• Verification data: Lack of detailed experimental results and comparative data.

Future Directions

• Multi-source information fusion: Combine code comments, Issue history, commit logs, etc.
• Dynamic feedback: Adjust parameter priorities based on historical test results.
• Cross-project learning: Apply transfer learning to similar projects.