The repository of the SANER'21 paper titled "On the Impact of Flaky Tests in Automated Program Repair"

This repository contains the artefact of the paper “On the Impact of Flaky Tests in Automated Program Repair” under review by SANER2021.

• results:
  • RQ1: This table contains all the statistical data of flaky tests we find out from Defects4J benchmark where each line illustrates the commit date, commit id, java file name, as well as all the flaky methods in this test file. Note that the commit id is empty if there is no update about the java file in that line since last version.
    • flaky_methods.xlsx
  • RQ2: Here are the execution results of 10 times running of flaky tests under 4 different environment. Detailed execution log of each running are classified by project and named in the format test_log_[project_name].txt,the summarized data of each project are named in the format Flaky_test_[project_name].xlsx. The Flaky_test_all.xlsx contain the summarized data for all projects.
    • jdk1.7 + ubuntu16.04
    • jdk1.7 + ubuntu18.04
    • jdk1.8 + ubuntu16.04
    • jdk1.8 + ubuntu18.04
  • RQ3: This folder contains execution results of the fault localization tool GZoltar-V1.7 under two jdk versions. Result of each bug is named in the format [project_name]_[bug_id].csv.
    • gzoltar_reuslt_flaky_jdk1.7
    • gzoltar_reuslt_flaky_jdk1.8
    • gzoltar_reuslt_noFlaky_jdk1.7
    • gzoltar_reuslt_noFlaky_jdk1.8
  • RQ4: This folder contains repair results of different APR tools. For the 10 tools from RepairThemAll framework, each result is named in format validation_result_[APR_name].txt  which indicates whether the previous patch can be generated this time. For TBar, we release all the generated patches including fixed and partially fixed.
    • RepairThemAll
    • TBar