Can AI Agents Write Good Property Tests? A Replication Experiment

This project replicates the paper 'Can Large Language Models Write Good Property Tests?' and extends it to the context of AI agents, comparing AI-generated and human-written property testing strategies. Key tools include Python's Hypothesis property testing library and the OpenAI Codex model, aiming to explore the potential and limitations of AI-assisted software testing.

Software testing is key to ensuring code quality. As an emerging paradigm, property testing focuses on the general properties of code rather than specific inputs and outputs. A 2023 related paper explored the ability of LLMs to generate property tests, and this project extends the scenario to the context of AI agents. Traditional unit tests verify specific cases (e.g., add(2,3)=5), while property tests use Hypothesis to generate random inputs to verify general rules (e.g., the commutative law of addition), which can uncover boundary cases that are hard for humans to think of.

Project objectives include: 1. Replicate the original paper using the same prompts and verify the effect in the AI agent context; 2. Compare AI-generated and human-written testing strategies; 3. Integrate Hypothesis and Codex tools. The experiment selects representative code snippets such as data structure operations and mathematical calculations, and evaluates them from the dimensions of coverage, effectiveness, readability, and completeness. The comparison benchmarks are human-written tests and basic AI-generated tests without agent optimization.

Hypothesis plays the core role in test execution: generating random inputs that meet constraints, executing tests and capturing failure cases, and providing detailed error reports. In the AI agent context, Codex can iteratively improve tests based on execution feedback, self-correct using error information, and interact with toolchains (type checkers, test runners) to generate property test code.

The advantages of AI include quickly generating initial test drafts, identifying common property patterns, and diverse testing strategies; challenges include accurately understanding code intent, covering special boundary cases, and avoiding false pass tests. The most effective model is human-AI collaboration: AI generates candidate tests, and humans review and refine them.

Implications of this project: Test design is moving towards automation; developers need to shift to high-level strategy design in collaboration with AI; AI-assisted testing will become a standard component of CI/CD. Limitations include limited scale and complexity of test code, insufficient coverage of areas like concurrency, and subjective evaluation criteria. Future directions can extend to complex codebases, explore multi-round conversational test refinement, and combine formal verification with AI-generated tests.