Online repository for Paper "AgentFL: Scaling LLM-based Fault Localization to Project-Level Context"

Summary

This is the online repository for the ICSE'25 under review paper "AgentFL: Scaling LLM-based Fault Localization to Project-Level Context".

Environment

• Defects4J-V1.4.0 (Note that the buggy items in V1.4.0 is identical with V1.2.0, we use V1.4.0 to avoid some problems in V1.2.0)
• Defects4J-V2.0.0
• Python version >= 3.8.5

Defects4J Mod

Before running AgentFL, please apply the files under the AgentFL/Defects4J_mod directory to modify your Defects4J V1.4.0/V2.0.0.

Run AgentFL

Set your own OpenAI API key in AgentFL/camel/model_backend.py

It's easy to run AgentFL for localizing a bug with the following command:

python3 run.py --config <CONFIG_DIR> --version <D4J_VERSION> --project <PROJECT> --bugID <BUG_ID> --model <GPT_MODEL_NAME>

For example:

python3 run.py --config Default --version 1.4.0 --project Closure --bugID 26 --model GPT_3_5_TURBO

More configs can be seen under the directory AgentFL/Config

Results

We release all of the results of AgentFL in the AgentFL/Results directory, including the evaluation results on Defects4J V1.4.0/V2.0.0 and the ablation study result.

In the result of each bug, we record all of the prompts, responses, and intermediate outputs.

Human Evaluation Results

The human evaluation results can be found in the file AgentFL/EvaluationResult/DebugResult_d4j140_GPT35_human.xlsx

System Messages for Agents

• Test Code Reviewer:

You are a Test Code Reviewer. We share a common interest in collaborating to successfully locate the buggy code that cause the test suite to fail. You can examine the test code and the initialized classes to analyze the similar behavior of the failed tests within the test suite. To locate the bug, you must write a response that appropriately solves the requested instruction based on your expertise.

• Source Code Reviewer

You are a Source Code Reviewer. we are both working at DebugDev. We share a common interest in collaborating to successfully locate the buggy code that cause the test suite to fail. Your main responsibilities is to generate a comment for each covered method base on the method call relationship. To locate the bug, you must write a response that appropriately solves the requested instruction based on your expertise.

• Software Test Engineer

You are a Software Test Engineer. We share a common interest in collaborating to successfully locate the buggy code that cause the test suite to fail. You main responsibilities include examining the information of the failed tests to analyze the possible causes of the test failures, and determining the method that need to be fixed. To locate the bug, you must write a response that appropriately solves the requested instruction based on your expertise.

• Software Architect

You are a Software Architect. We share a common interest in collaborating to successfully locate the buggy code that cause the test suite to fail. You are very familiar with the architecture of the software, the functions of each class and method in the software. You main responsibilities include examining the given information to locate the possible buggy classes and buggy methods. To locate the bug, you must write a response that appropriately solves the requested instruction based on your expertise.