# Kick-the-Tires

First, change directory:
```
cd /PathToThisArtifact/compdiff-aflpp/scripts/
```
Then run:
```
sudo ./run.sh captainrc
```
If you do'nt see any errors after 10 seconds, it's successful and type `CTRL+C` to terminate the command. Then remove temporary files:
```
sudo rm -rf ../experiments/
```

Next, we need to use the provided docker image.
To import the docker image:
```
cat compdiff-image.tar | docker import - compdiff_artifact
```

Start an interactive session:
```
docker run --name compdiff_container -it compdiff_artifact bash
```

Run the second kick-the-tires script:
```
./kick_the_tires_2.py
```


# Abstract

The docker image contains the code we used for our experiments as well as scripts to generate the numbers, tables, and figures in our paper.

To import the docker image:
```
cat compdiff-image.tar | docker import - compdiff_artifact
```

Start an interactive session:
```
docker run -it compdiff_artifact bash
```


# Validating the Statistics in the Paper

Note that, all the evaluations should be done within the docker container.
In this section, we provide detailed instructions to validate the important data used in the paper. 
We first validate the data shown in Sections 4.1 and 4.2, where the Juliet testsuite is used. Then, we validate Section 4.3, where our CompDiff-AFL++ is used to fuzz 23 real-world programs.

## Validating Table 3

We use the Juliet testsuite C/C++ (https://samate.nist.gov/SARD/test-suites/112) for evaluation. We have put the unzipped Juliet testsuite under `/compdiff/dataset/juliet/`. In the following steps, we will reproduce all data shown in Table 3.

### Sanitizers columns in Table 3

We first reproduce the "Sanitizers" columns in Table 3.

**1. Compile**

First, compile all the Juliet testcases with ASAN, UBSAN, and MSAN. The whole compilation takes ~0.5 hours in a 32-core machine:

```
cd /compdiff/juliet/sanitizers
./compile_sanitizers.py
```

You will see the compilation details in the terminal. The resulting binaries are in `/compdiff/juliet/sanitizers/bin/`. Each subdirecory contains:

- `C-asan` : all testcases compiled with `clang -fsanitize=address`.
- `C-ubsan` : all testcases compiled with `clang -fsanitize=undefined`.
- `C-msan` : all testcases compiled with `clang -fsanitize=memory`.

**2. Run sanitizers**

Then, use each sanitizer to detect undefined behaviors. This will take ~10min in a 32-core machine:
```
./run_sanitizers.py
```
This script will execute sanitizer-enabled binaries and log the results. The results are saved as `./results/CWE-XXX-SAN.txt`. There are three digits in each row, indicating respectively that whether or not ASAN, UBSAN, and MSAN can detect undefined behavior in the testcase. For example,
```
tail -n 1 ./results/CWE121_Stack_Based_Buffer_Overflow-SAN.txt
```

prints

```
testcases/CWE121_Stack_Based_Buffer_Overflow/s01/CWE121_Stack_Based_Buffer_Overflow__CWE129_large_68.out:1:1:0
```

The last digits `1:1:0` indicates that ASan and UBSan detected undefined behavior in the testcase while MSan did not.

**3. Generate data**

Finally, we generate the "Sanitizers" data in Table 2:

```
./generate_data_sanitizers.py
```

Since there is a pre-set timeout threshold, you may observe ±5% differences due to hardware difference.

### CompDiff columns in Table 3

We now reproduce the last two columns in Table 3. They are the bug detection results of our proposed CompDiff.

**1. Compile**

First, compile all the Juliet testcases with 10 different compiler configurations. The whole compilation takes ~1.5 hours in a 32-core machine:

```
cd /compdiff/juliet/compdiff
./compile_compdiff.py
```

You will see the compilation details in the terminal. This command will compile each Juliet testcase with 10 different compiler configurations. The resulting binaries are in `/compdiff/juliet/compdiff/bin/`. Each subdirecory contains:

- `C-gcc-0` : all testcases compiled with `gcc -O0`.
- `C-gcc-1` : all testcases compiled with `gcc -O1`.
- `C-gcc-2` : all testcases compiled with `gcc -O2`.
- `C-gcc-3` : all testcases compiled with `gcc -O3`.
- `C-gcc-s` : all testcases compiled with `gcc -Os`.
- `C-clang-0` : all testcases compiled with `clang -O0`.
- `C-clang-1` : all testcases compiled with `clang -O1`.
- `C-clang-2` : all testcases compiled with `clang -O2`.
- `C-clang-3` : all testcases compiled with `clang -O3`.
- `C-clang-s` : all testcases compiled with `clang -Os`.

**2. Run CompDiff**

Then, use CompDiff to detect undefined behaviors (~15min in a 32-core machine):
```
./run_compdiff.py
```
This script will execute binaries compiled with different compiler configurations, cross-check their outputs, and log inconsistencies. The results are saved as `./results/CWE-XXX-CompDiff.txt`. 

Each row in a text file represents whether or not CompDiff detected undefined behavior in a testcase. For example,
```
head -n 1 ./results/CWE121_Stack_Based_Buffer_Overflow-CompDiff.txt
```

prints

```
testcases/CWE121_Stack_Based_Buffer_Overflow/s03/CWE121_Stack_Based_Buffer_Overflow__CWE805_char_declare_memmove_03.out:1
```

The last digit `1` indicates that CompDiff has detected undefined behavior in the testcase, otherwise it would be `0`.

**3. Generate data**

Finally, we generate the CompDiff data in Table 2:

```
./generate_data_compdiff.py
```

Since there is a pre-set timeout threshold, you may observe ±5% differences due to hardware difference. In "CompDiff-Unique" of the result, you may observe ±100 difference in the first row. This is normal since the total number of tests is ~3,000 where a small percentage difference can lead to a large number.

### Coverity columns in Table 3

Since [Coverity](https://scan.coverity.com/) is close-sourced, we cannot run its detection results by ourself. However, its [official website](https://scan.coverity.com/projects/juliet_test_suite_v1-3_for_c_cpp2) provides detection results on the Juliet testsuite. We have downloaded and put it under `/compdiff/juliet/coverity/Outstanding+Issues.csv`.

To extract the detection result of Coverity:
```
cd /compdiff/juliet/coverity
./extract_coverity.py
```

This script will extract the detection result into `./coverity_result.json`. The following command will analyze the result and print it:
```
./print_coverity.py
```

Since there is a pre-set timeout threshold, you may observe ±5% differences due to hardware difference.

### CppCheck columns in Table 3

We use [CppCheck-2.9](https://sourceforge.net/projects/cppcheck/files/cppcheck/2.9/) for evaluation. It has been installed under `/compdiff/juliet/cppcheck/cppcheck-2.9/build`.
To run CppCheck on the Juliet testsuite (~1h):

```
cd /compdiff/juliet/cppcheck
./run_cppcheck.py
```

This script will invoke `cppcheck` tool to detect undefined behaviors. The results will be saved in `./results/`. 
Next, extract CppCheck results:

```
./extract_cppcheck.py
```

This script will extract the detection results into `./cppcheck_result.json`. The following command will analyze the result and print it:
```
./print_cppcheck.py
```

Since there is a pre-set timeout threshold, you may observe ±5% differences due to hardware difference.

### Infer colomns in Table 3

We use [Infer-v1.1.0](https://fbinfer.com/) for evaluation. It has been installed under `/compdiff/juliet/infer/infer-v1.1.0`. 
To run Infer on the Juliet testsuite (~1h):

```
cd /compdiff/juliet/infer
./run_infer.py
```

This script will invoke `infer` tool to detect undefined behaviors. The results will be saved in `./results/`.
Next, extract the results:

```
./extract_infer.py
```

This script will extract the detection results into `./infer_result.json`. The following command will analyze the result and print it:
```
./print_infer.py
```

Since there is a pre-set timeout threshold, you may observe ±5% differences due to hardware difference.

## Validating Table 2

Although Table 2 is presented before Table 3 in the paper, it depends on the results of Table 3 as we need to first run all testcases and filter out timed-out ones.
To obtain the statistics in Table 2:

```
cd /compdiff/juliet
./benchmark_statistics.py
```

This script will count the number of testcases used in `/compdiff/juliet/*` and filter out all timed out ones.

## Validating Figure 1

We reuse the meta-data generated above when evaluation "CompDiff" columns. To get the result of each subset of compilers:

```
cd /compdiff/juliet/subset
./extract_subset.py
```

This script will count how many bugs each subset of compilers can detect and then save the result in `./compdiff_subset.json`.
To obtain Figure 1:
```
./plot_subset.py
```

This script will generate `./comdiff_subset.png`, which is identical to Figure 1.

## Validating Table 5

We used CompDiff-AFL++ to fuzz 23 real-world programs. Since we run each fuzzing procedure 24 hours and repeated 10 times, it may need more than 1 CPU year to regenerate all bug reports. In this artifact, we provide all bugs found by CompDiff-AFL++ and then show how to use CompDiff-AFL++ to fuzz a target.

First, change working directory:
```
cd /compdiff/real
```

The json file `./bugs.json` contains details of all bug reports. We provide a script `analyze_bugs.py` to emit useful information from this json file:
- `./analyze_bugs.py --list-programs` prints all programs.
- `./analyze_bugs.py --list-bugs` prints all bug details.
- `./analyze_bugs.py --program tcpdump` prints bug details of `tcpdump`.

Using `bugs.json` we can also generate Table 5:
```
./summary_bugs.py
```

## Validating Table 6

As stated in the paper (RQ3 in page 10), data used for Table 6 are manually generated. Thus, we do not provide instructions here. One can get this information by querying each bug report listed by `./analyze_bugs.py --list-bugs`.

## Validating Figure 2

The json file `./compdiff_real.json` contains the information that for each subset of compilers, if a bug is detecable of not with CompDiff. The following script will use this file to generate Figure 2:
```
./plot_compdiff_subset.py
```

The generated image `compdiff_subset_real.png` is identical to Figure 2.

## Fuzzing a target with CompDiff-AFL++

We here provide instructions on fuzzing a target with CompDiff-AFL++. As explained before, we cannot fuzz all targets due to time constraints and thus we only show an example.

You need to exit/detach from the running docker container (by `CTRL p + q` or typing `exit`).

Change directory to `/PathToThisArtifact/compdiff-aflpp`. Here you can find a few directories:
- `fuzzers/compdiff_aflpp` contains the sourcecode of CompDiff-AFL++ and scripts used to build, instrument, and fuzz a target.
- `targets/` contains all 23 targets including input seeds and build script.
- `scripts/` contains useful scripts for fuzzing with CompDiff-AFL++.

Suppose we want to fuzz the target `mujs` with CompDiff-AFL++:

1. `cd scripts/`
2. Edit `vim captainrc`: add `mujs` in the field "TARGETS" (it is already there)
3. Modify other fields: "OUT_DIR" for the output dirname; "SLAVES" for the number of parallel fuzzing processes; "REPEAT" for repetition times; "TIMEOUT" for timeout value in seconds; "CPU_i" for the starting CPU ID.
4. Run `sudo ./run.sh captainrc`. You need to enter sudo password as it needs to mount tpmfs.
This command will build `mujs` with CompDiff-AFL++ compilers and then fuzz it. You will see the following message printed (may need ~10min to see the full message):
```
[sudo] password for xxx:
Building diffuzz/compdiff_aflpp/mujs
Container compdiff_aflpp/mujs/0 started on CPU 0
Container compdiff_aflpp/mujs/1 started on CPU 2
Waiting for jobs to finish
```

Now CompDiff-AFL++ is fuzzing `mujs` and it would stop until reaching the pre-set timeout value. You may type `CTRL+C` to stop fuzzing. The fuzzing result will be saved in `/PathToThisArtifact/compdiff-aflpp/experiments/`. You may need sudo previledge to access the results. Due to the stochastic nature of fuzzing, we do not anticipate that you could observe any bug in a short period of time.