Ciallo: An Automated Program Repair Framework Based on Prompt Chaining and Large Language Models

Ciallo is an innovative Automated Program Repair (APR) framework whose core lies in combining the code generation capabilities of Large Language Models (LLM) with Prompt Chaining technology to locate and fix software defects through a structured multi-stage reasoning process. It addresses the limitations of traditional APR methods (such as insufficient template matching or search) and the flaws of pure LLM single-round dialogue (context constraints, insufficient reasoning depth, etc.). This article will introduce the features and value of Ciallo from aspects like background, design, technical architecture, experimental verification, deployment and usage.

Software defect repair is a time-consuming and error-prone part of development; about 50% of development time in large projects is spent on debugging. Traditional APR technologies are limited by template matching or search-based methods, making it difficult to handle complex logical errors. In recent years, LLM has shown outstanding performance in code understanding and generation. Ciallo is based on this trend and proposes a new APR framework combining prompt chaining and LLM to address the challenges of traditional methods.

The name Ciallo is derived from the abbreviation of "prompt chaining and large language models". Its core design philosophy is to guide LLM multi-stage reasoning through structured prompt chaining instead of generating repair solutions in one go. Compared to single-round dialogue, prompt chaining has the following advantages: 1. Solving context window limitations; 2. Guiding deep structured reasoning; 3. Improving result controllability. By decomposing complex tasks into consecutive subtasks, the output of each stage serves as the input for the next, forming a clear reasoning chain.

The technical architecture of Ciallo includes three main components:

1. AST Difference Analysis: Uses the GumTree Spoon AST Diff tool, which has the advantages of semantic awareness, fine granularity, and noise filtering;
2. Multi-Stage Prompt Chaining: Divided into four stages: defect localization (analyzing failed tests to locate suspicious code), root cause analysis (identifying root causes and repair strategies), patch generation (generating patches in diff format), and patch verification (running tests to validate repairs);
3. Plausible Patch Screening: Filters valid patches through syntax checks, test pass rate verification, and regression tests, and stores them in a specified directory.

Ciallo is validated using Defects4J (a standard dataset for APR research, containing defects from real Java projects). Evaluation metrics include repair rate, plausible patch rate, correct patch rate, and average repair time. Compared to traditional APR tools (such as GenProg) and pure LLM methods, Ciallo shows advantages of higher repair rate, fewer incorrect repairs, and better interpretability.

Ciallo supports Dockerized deployment to simplify environment configuration. Quick start steps:

1. Clone the repository;
2. Download the AST difference analysis tool;
3. Configure the OpenAI API key;
4. Pull the Docker image;
5. Start the container to run. Parameters such as model selection, output directory, and prompt templates can be configured via prog_params.py.

Three major challenges were addressed during the development of Ciallo:

1. Token Counting and Cost Control: Uses GPT-4o embedding token counting as the benchmark, records usage in each stage, and provides cost estimates;
2. Context Length Limitation: Addressed through code slicing, hierarchical summarization, and iterative retrieval;
3. Unstable Patch Quality: Improved stability through temperature tuning (e.g., 0.2), multi-sampling strategy, and post-processing verification.

Ciallo's application scenarios include automatic repair in CI/CD pipelines, code review assistance, legacy code maintenance, and educational research platforms. Limitations include language support (currently mainly Java), efficiency in large-scale projects, and ability to repair defects in complex domains. Future directions: Introduce RAG technology, multi-agent collaboration architecture, and reinforcement learning to optimize prompt chaining strategies.