Extended Empirical Study on Large Language Models for Multilingual Equivalent Mutant Detection

This study systematically evaluates the ability of various large language models (including GPT-4, DeepSeek-Coder, CodeLlama, Qwen2.5-Coder, etc.) to detect equivalent mutants across multiple programming languages, providing important references for mutation testing automation in the software testing field. The study covers core content such as background, models, methods, findings, applications, and conclusions.

Mutation testing is a key technique in software testing to evaluate the effectiveness of test cases, but equivalent mutants (mutants with the same semantics as the original program) need to be manually identified, consuming a lot of resources. With the breakthrough of large language models in code understanding tasks, this study aims to systematically evaluate the ability of mainstream LLMs to detect equivalent mutants in multilingual environments.

The study covers multiple types of models: general-purpose large language models (GPT-4, GPT-3.5-Turbo, Llama3), code-specific models (DeepSeek-Coder, CodeLlama, StarCoder, Qwen2.5-Coder), encoder-decoder architecture models (CodeBERT, GraphCodeBERT, CodeT5, etc.), and embedding models (Text-Embedding series).

A multi-dimensional evaluation framework is adopted: 1. Dataset construction: Organize multilingual code samples and corresponding mutants; 2. Experimental design: Independent experiment directories for each model, including specific configurations and evaluation scripts; 3. Manual benchmark: Manually annotated results serve as the gold standard for model accuracy.

1. Significant differences in model capabilities: Code-specific models are usually superior to general-purpose large language models; 2. Challenges in multilingual support: High program semantic understanding ability is required; 3. Prompt engineering affects judgment accuracy, including strategies such as zero-shot, few-shot, and chain-of-thought.

Practical significance: Provide an empirical basis for automated equivalent mutant detection tools and reduce manual review workload; Model selection guidance: Models such as CodeT5 and UniXCoder are more cost-effective in equivalence judgment. Future research directions: Explore the performance of large-scale models, multimodal methods, and language-specific detectors.

This study provides valuable insights into the application potential of LLMs in the software testing field, and automated equivalent mutant detection is moving from theory to practice. The research code and dataset have been open-sourced, providing a reproducible basis for subsequent studies.