ESEC/FSE 2023 Artifact for "Heterogeneous Testing for Coverage Profilers Empowered with Debugging Support"

Artifacts for "Heterogeneous Testing for Coverage Profilers Empowered with Debugging Support"

Comparison for DeCov, C2V, and Cod

Decov: Heterogeneous Testing for Coverage Profilers Empowered with Debugging Support

C2V: Hunting for bugs in code coverage tools via randomized differential testing

Cod: Automatic Self-Validation for Code Coverage Profilers

Table of Contents

• Background
• Install
• Usage
• Data

Background

We re-run Decov, C2V, and Cod for Gcov-7 and LLVM-cov-6.0 along with Gcc-7 and LLVM-6.0 over the test programs in the Gcc-7.4.0 testsuites for the comparison.

Install

This project uses gcc-7, gdb, llvm-6.0 and lldb. Go check them out if you don't have them locally installed.

$ apt install python3 gcc-7 gdb llvm-6.0 clang-6.0 lldb-6.0

Usage

First, download the gcc-7.4.0.tar.xz:

$ wget http://ftp.gnu.org/gnu/gcc/gcc-7.4.0/gcc-7.4.0.tar.xz 

Then unzip the downloaded file:

$ tar -xvf gcc-7.4.0.tar.xz 

Run C2V, Cod, and Decov over the testsuite of gcc-7.4.0, respectively:

$ python3 main.py --source="gcc-7.4.0/gcc/testsuite/" --method='C2V'
$ python3 main.py --source="gcc-7.4.0/gcc/testsuite/" --method='COD'
$ python3 main.py --source="gcc-7.4.0/gcc/testsuite/" --method='DECOV'

Note that, we recommend to use the parallel option when using our tool if your computer has multi-cores:

$ python3 main.py --parallel --source="gcc-7.4.0/gcc/testsuite/" --method='C2V'
$ python3 main.py --parallel --source="gcc-7.4.0/gcc/testsuite/" --method='COD'
$ python3 main.py --parallel --source="gcc-7.4.0/gcc/testsuite/" --method='DECOV'

Data

Experimental data will be stored in the Expr folder:

C2V

• C2V/files-DIFFCOV-C2V.txt: the filenames when the coverage statistics is inconsistent between Gcov and LLVM-cov
• C2V/files-SAMECOV-C2V.txt: the filenames when the coverage statistics is consistent between for Gcov and LLVM-cov
• C2V/lines-DIFFCOV-C2V-FULL.txt: the filenames and line numbers where the coverage statistics is inconsistent between Gcov and LLVM-cov
• C2V/lines-SAMECOV-C2V-FULL.txt: the filenames and line numbers where the coverage statistics is consistent between Gcov and LLVM-cov

Cod

• COD/files-DIFFCOV-COD-gcov.txt and COD/files-DIFFCOV-COD-llvm-cov.txt: the filenames when the coverage statistics of the original program and the equivalent mutant violate the metamorphic relations for Gcov and LLVM-cov
• COD/files-SAMECOV-COD-gcov.txt and COD/files-SAMECOV-COD-llvm-cov.txt: the filenames when the coverage statistics of the original program and the equivalent mutant are consistent for Gcov and LLVM-cov
• COD/lines-DIFFCOV-COD-gcov.txt and COD/lines-DIFFCOV-COD-llvm-cov.txt: the filenames and line numbers where the coverage statistics of the original program and the equivalent mutant violate the metamorphic relations for Gcov and LLVM-cov
• COD/lines-SAMECOV-COD-gcov.txt and COD/lines-SAMECOV-COD-llvm-cov.txt: the filenames and line numbers where the coverage statistics of the original program and the equivalent mutant are consistent for Gcov and LLVM-cov

Decov

• DECOV/files-DIFFCOV-DECOV-gcov.txt and DECOV/files-DIFFCOV-DECOV-llvm-cov.txt: the filenames when the coverage statistics from coverage profiler is inconsistent with the hit count from the debugger for Gcov and LLVM-cov.
• DECOV/files-SAMECOV-DECOV-gcov.txt and DECOV/files-SAMECOV-DECOV-llvm-cov.txt: the filenames when the coverage statistics from coverage profiler is consistent with the hit count from the debugger for Gcov and LLVM-cov.
• DECOV/lines-DIFFCOV-DECOV-gcov.txt and DECOV/lines-DIFFCOV-DECOV-llvm-cov.txt: the filenames and line numbers where the coverage statistics from coverage profiler is inconsistent with the hit count from the debugger for Gcov and LLVM-cov.
• DECOV/lines-SAMECOV-DECOV-gcov.txt and DECOV/lines-SAMECOV-DECOV-llvm-cov.txt: the filenames and line numbers where the coverage statistics from coverage profiler is consistent with the hit count from the debugger for Gcov and LLVM-cov.