This repository contains the replication package for the paper "The Art of Repair: Optimizing Iterative Program Repair with Instruction-Tuned Models" by Fernando Vallecillos Ruiz, Max Hort, and Leon Moonen, accepted for the research track of the 29th International Conference on Evaluation and Assessment in Software Engineering (EASE 2025). A preprint of the paper is included.

The source code is distributed under the MIT license, and except for 3rd party datasets that come with their own license, all documentation, data, models and results in this repository are distributed under the CC BY 4.0 license.

Repository Overview

This repository contains the necessary scripts, data, and resources to replicate the experiments presented in our conference paper. The structure of this repository has been organized to facilitate ease of use for researchers interested in reproducing our results, conducting similar analyses, or building upon our work.

Repository Structure

* Note that all contents except for the model files from the models/ folder are included in the compressed zip file in this Zenodo repository. The model files are uploaded separately to the repository to facilitate individual downloads, as several of them are relatively large (9.5-11.2GB).

Detailed Folder Descriptions

Analysis (analysis/)

This folder contains Jupyter notebook scripts used to generate tables and visual analyses of the experimental data. These scripts are designed to assist in visualizing results, comparing performance metrics, and summarizing experimental outcomes. Researchers can easily export the generated tables to spreadsheets for further processing or visualization. The outputs help in validating the experiment's consistency and provide insights into the performance of various model configurations.

Inference and Validation Source (inference_and_validation_src/)

The Python scripts in this folder are used for generating patches and validating them against predefined benchmarks. We utilize the "Fire" library to parse parameters and execute the relevant methods efficiently. This folder contains:

• Scripts for generating patches directly from the benchmark data or using iterative approaches.
• Validation utilities for Defects4J and HumanEval benchmarks to ensure the generated patches are functional and comply with benchmark requirements.

Key components include:

• Patch generation logic.
• Validation commands for HumanEval and Defects4J benchmarks.
• Utilities to verify data integrity of generated JSON files.

Training Source (train_src/)

This folder contains the scripts used for model fine-tuning:

• full_finetune.py: This script performs full fine-tuning of a model on a given training dataset. It updates all trainable parameters to achieve optimal model performance on the target task.

• lora_finetune.py: This script implements LoRA (Low-Rank Adaptation) fine-tuning. LoRA is a parameter-efficient fine-tuning approach where only a smaller subset of model parameters are updated, making it effective for resource-constrained tasks.

Inference Scripts (inference_scripts/)

These Bash scripts are designed to automate the inference process by submitting multiple iterations of inference and validation jobs to the compute cluster. The scripts create job dependencies, ensuring that all necessary tasks are completed in a logical sequence.

The available inference scripts include:

• model_inferencing_adjustable_FULL_d4j_big.sh: Executes inference for specified model configurations with multiple iterations and outputs per iteration.
• model_inferencing_adjustable_FULL_d4j_lora_big.sh: Similar to the previous script, but optimized for LoRA-based models.

These scripts accept three parameters:

• MODEL: The name of the model, as found in the models/ folder.
• NUM_ITERATIONS: The number of iterations to run.
• NUM_OUTPUTS: The number of outputs generated in each iteration.

Citation and Zenodo links

We hope this package serves as a useful resource for reproducing and expanding upon our research results. Please cite this work by referring to the published paper:

Fernando Vallecillos Ruiz, Max Hort, and Leon Moonen, 2025. The Art of Repair: Optimizing Iterative Program Repair with Instruction-Tuned Models. In proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering (EASE 2025), ACM, 12 pages.

@inproceedings{ruiz2025:art,
    title = {{The Art of Repair: Optimizing Iterative Program Repair with 
              Instruction-Tuned Models}},
    author = {Ruiz, Fernando Vallecillos and Hort, Max and Moonen, Leon},
    booktitle = {{Proceedings of the 29th International Conference on Evaluation 
                  and Assessment in Software Engineering (EASE)}},
    year = {2025},
    pages = {12},
    publisher = {{ACM}},
    language = {en}
}

The replication package is archived on Zenodo with DOI: 10.5281/zenodo.15294695.