Artifacts for Code Coverage Profiler Validation with Debugging Support

Code Coverage Profiler Validation with Debugging Support

Table of Contents

• Background
• Install
• Usage
• Experiment
• Violations
• Bugs

Background

DeCov is the tool for code coverage profiler validation with debugging support. We totally submit 24 bug reports for gcov and llvm-cov. Of these 24 bug reports, 18 are confirmed/fixed by developers, 2 are marked as duplicate, and the other 4 bug reports are waiting for the developers' confirmation.

Install

This project uses gcc, gdb, llvm and lldb. Go check them out if you don't have them locally installed.

apt install gcc gdb llvm lldb

Usage

To use DeCov for gcov validation:

python3 decov-gcov-pexpect.py -s test.c

or

python3 decov-gcov-pexpect.py -s testprograms

To use DeCov for llvm-cov validation:

python3 decov-llvmcov-pexpect.py -s test.c

or

python3 decov-llvmcov-pexpect.py -s testprograms

Experiment

First, download the gcc-10.1.0.tar.xz:

wget http://ftp.gnu.org/gnu/gcc/gcc-10.1.0/gcc-10.1.0.tar.xz

Then, unzip gcc-10.1.0:

tar -xvf gcc-10.1.0.tar.xz

Finally, run Decov over the testsuite of gcc-10.1.0:

python3 decov-gcov-pexpect.py -s gcc-10.1.0/gcc/testsuite/

or

python3 decov-llvmcov-pexpect.py -s gcc-10.1.0/gcc/testsuite/

Note that, we recommend to use the parallel option when using our tool if your computer has multi-cores:

python3 decov-gcov-pexpect.py --parallel -s gcc-10.1.0/gcc/testsuite/

or

python3 decov-llvmcov-pexpect.py --parallel -s gcc-10.1.0/gcc/testsuite/

Violations

Experimental data are in the Expr folder:

• files-DIFFCOV-gcc.txt and files-DIFFCOV-clang.txt list the filenames in the testsuites of GCC-10.1.0 that the coverage statistics from coverage profilers inconsistent with the hit count from debuggers
• DIFFCOV-Gcov-vs-GDB.txt and DIFFCOV-LLVMcov-vs-LLDB.txt list the filesname and the the respected lines of code that violating HMR, coverage statistics is not consistent with the hit count
• files-SAMECOV-gcc.txt and files-SAMECOV-clang.txt list the filenames in the testsuites of GCC-10.1.0 that the coverage statistics from coverage profilers consistent with the hit count from debuggers.
• files-CPLERROR-gcc.txt and files-CPLERROR-clang.txt list the filenames in the testsuites of GCC-10.1.0 that cannot be correctly compiled independently by respected compilers

Bugs

Bugs reported by DeCov for LLVM-cov:

llvm-cov 48818 Status History: "NEW"

llvm-cov 48795 Status History: "NEW"

llvm-cov 48278 Status History: "NEW" -> "DUPLICATE"

llvm-cov 48264 Status History: "NEW" -> "INVALID" -> "REOPENED"

llvm-cov 48263 Status History: "NEW"

llvm-cov 45850 Status History: "New" -> "CONFIRMED"

llvm-cov 45849 Status History: "New" -> "FIXED" -> "REOPENED" -> "FIXED"

llvm-cov 45846 Status History: "NEW" -> "CONFIRMED"

llvm-cov 45845 Status History: "NEW" -> "CONFIRMED"

llvm-cov 45844 Status History: "NEW" -> "DUPLICATE"

llvm-cov 45821 Status History: "NEW" -> "CONFIRMED"

llvm-cov 45678 Status History: "NEW" -> "CONFIRMED"

llvm-cov 45595 Status History: "NEW" -> "CONFIRMED"

Bugs reported by DeCov for Gcov:

gcov 97925 Status History: "UNCONFIRMED" -> "NEW"

gcov 97924 Status History: "UNCONFIRMED" -> "NEW"

gcov 97923 Status History: "UNCONFIRMED" -> "NEW"

gcov 97917 Status History: "UNCONFIRMED" -> "NEW"

gcov 97910 Status History: "UNCONFIRMED" -> "NEW"

gcov 93757 Status History: "UNCONFIRMED" -> "NEW"

gcov 93735 Status History: "UNCONFIRMED" -> "NEW"

gcov 93726 Status History: "UNCONFIRMED" -> "WAIT"

gcov 93725 Status History: "UNCONFIRMED" -> "NEW"

gcov 93706 Status History: "UNCONFIRMED" -> "NEW"

gcov 93680 Status History: "UNCONFIRMED" -> "NEW"