Reproduction Package for STTT 2024 Submission `FaultY: A Framework for Managing Fault Candidates'

FaultY

This framework allows managing the output of fault localization techniques. The artifact consists of two parts:

1. running our implementations of the algorithms on some tasks.
2. showcase the import of faults.

The VM

The username is sosylab. The password is sosy.

We use VirtualBox Graphical User Interface Version 6.1.38_Ubuntu r153438 to import the VM. The VM runs Ubuntu 22.04 (64 bit). We recommend 8 GB of RAM and 2 CPU cores. The VM uses cpachecker on revision 42446 on trunk. The VM uses benchexec 3.15-dev.

Files

The Desktop contains symlinks to 4 files and 1 directory called faulty. The folder faulty contains 4 directories and 8 files.

Directories

• benchexec: This tool allows us to execute a certain configuration of a tool on multiple input tasks.
• cpachecker: This tool contains our implementation of the framework
• results: After running our scripts, all results will be shown in this directory
• compare: Contains the JSON file before and after the import of faults when running import.sh in cpachecker/. Additionally, it contains the diff of the two files.

Files:

• execute_full_run.sh: Runs every configuration on a subset of programs and copies the output to results/. Requires root access. The password is sosy.
• extract.py: Used by execute.sh to extract the relevant output from the benchmark runs.
• packages.txt: Lists all packages we installed.
• setup.py: Used to provide the correct access rights for benchexec. Requires root access.
• Tutorial.md: Contains a full tutorial on how to use the framework. Learn how to import faults, how to run fault localization, and how to implement a new algorithm.
• Tutorial.html: Tutorial.md but as HTML.
• ReadMe.md: This file. Location: ~/faulty/ReadMe.md
• ReadMe.html: This file, but as HTML. Location: ~/faulty/ReadMe.html

Instructions

Generate a Single Report

Run ./run.sh <fl> within cpachecker/ to run one of the implemented fault-localization techniques. Replace <fl> with one of errinv, maxsat, maxsatorg, distanceMetric, tarantula, ochiai, dstar to execute the respective configuration. The complete output can be found in cpachecker/output/. Here, the two files Counterexample.2.html (visualized report) and faultlocalization.json (our format) are the most relevant. The recommended workflow for ErrorInvariants (errinv) is:

cd ~/faulty/cpachecker
./run.sh errinv
open ~/faulty/cpachecker/output/Counterexample.2.html
gedit ~/faulty/cpachecker/output/faultlocalization.json

The command also prints the executed cpachecker commands in red. We expect that Counterexample.2.html contains a counterexample, where error-prone statements are marked red. Clicking on Change view displays more details about the fault localization run. For more information, see “Interpret the Data”

Generate the JSON File for Imports

Run ./import.sh <fl> within cpachecker/ to export a fault by internally calling ./run.sh <fl>. Replace <fl> with one of errinv, maxsat, maxsatorg, distanceMetric, tarantula, ochiai, dstar to run the respective configuration. Afterwards, the script imports the fault and applies two rankings and an explanation. This showcases how our framework can be used by external tools. The output can be found in compare/. It contains the JSON file before and after the import of faults. Additionally, it contains the diff of the two files. The recommended workflow for ErrorInvariants (errinv) is:

cd ~/faulty/cpachecker
./import.sh errinv
gedit ~/faulty/compare/diff.diff ~/faulty/compare/export.json ~/faulty/compare/import.json

The script also prints the executed cpachecker commands in red.

Generate a Set of Reports and JSON Files

Please make sure to execute the script from ~/faulty.

1. Execute execute_full_run.sh to run our implementations of fault localization techniques on predefined tasks.
2. As soon as the script finishes, the results can be found in the results/ directory. This may take some time (up to 15 min).
3. Double-click the HTML pages to take a look at the visualization of the faults (cf. Interpret the Data).
4. The exported version of the faults in our JSON format can also be found in the directory. They have the same base name.

The recommended workflow is:

cd ~/faulty/
./execute_full_run.sh
cd ~/faulty/results/
echo "Open all reports with Firefox"
firefox ~/faulty/results/*.html
echo "Open all exported JSONs in gedit"
gedit ~/faulty/results/*.json

Note that listing Current values in the HTML report is not supported as that would require an analysis of the program to find the values of each variable. The output format will not be extended to support values of variables, as it increases the complexity.

Please ignore the warnings of benchexec as we do not focus on measuring the time: - Unable to find pqos_wrapper, please install it for cache allocation and monitoring if your CPU supports Intel RDT (cf. https://gitlab.com/sosy-lab/software/pqos-wrapper) - LXCFS is not available, some host information like the uptime leaks into the container. - The TIMEOUT for MaxSatOrg is also expected.

The run will produce the file /home/sosylab/faulty/results/factorization_paper_example.maxsatorg.html. This file is the report of the running example in the paper.

Interpret the Data

A description of the JSON format can be found in the tutorial. See the next section for more details. The report contains two buttons with ‘?’. Clicking these buttons shows instructions on how to interpret the data. Change view toggles the view between the counterexample and the information of fault localization. The ranks of the lines are clickable and show the same information as seen after toggling the view. Every fault is displayed as a card. The header has 3 outlined text boxes. The first number (in green) shows the rank. The second number (in yellow) shows the suspiciousness score of the fault. The contents of the card display all additional information.

Optional Tutorial (Internet required)

The artifact for the paper ends here. To see how a new fault localization algorithm can be implemented, read the text below. A full tutorial on creating a new fault localization algorithm in cpachecker is available within cpachecker in ~/faulty/cpachecker/docs/tutorials/fault-localization. A symbolic link to the tutorial is provided next to this file. Internet access is required for building cpachecker after the changes to resolve the dependencies.