Static Stack-Preserving Intra-Procedural Slicing of WebAssembly Binaries

# About this artifact
This artifact contains the implementation and the results of the evaluation of a
static slicer for WebAssembly described in the ICSE 2022 paper titled "Static
Stack-Preserving Intra-Procedural Slicing of WebAssembly Binaries".

The artifact contains a docker image (`wassail-eval.tar.xz`) that contains
everything necessary to reproduce our evaluation, and the actual data resulting
from our evaluation:
1. The implementation of our slicer (presented in Section 4.1) is included in
   the docker machine, and is available publicly here:
   https://github.com/acieroid/wassail/tree/icse2022
2. Test cases used for our evaluation of RQ1 are included in the docker machine
   and in the `rq1.tar.xz` archive.
3. The dataset used in RQ2, RQ3, and RQ4 is included in the docker machine.
4. The code needed to run our evaluation of RQ2, RQ3, and RQ4 is included in the
   docker machine.
5. The scripts used to generate the statistics and graphs that are included in
   the paper for RQ2, RQ3, and RQ4 are included in the docker machine and as the
   `*.py` files in this artifact.
6. The data of RQ5 that has been used in our manual investigation is included in
   the docker machine and in the `rq5.tar.xz` archive, along with
   `rq5-manual.txt` detailing our manual analysis findings.

# How to obtain it
Our artifact is available on Zenodo at the following URL: https://zenodo.org/record/5821007

# Setting up the Docker image
## Downloading The Artifact
The artifact is available at the following URL: https://zenodo.org/record/5821007

## Loading The Docker Image
Once the artifact is downloaded in the file `icse2022slicing.tar.xz`, it can be extracted and loaded into Docker as follows (this takes a few minutes):
```
docker import icse2022slicing.tar.xz
```
To simplify further commands, you can tag the image using the printed sha256 hash of the image: if the `docker import` command resulted in the hash `54aa9416a379a6c71b1c325985add8bf931752d754c8fb17872c05f4e4b52ea2`, you can run:
```
docker tag 54aa9416a379a6c71b1c325985add8bf931752d754c8fb17872c05f4e4b52ea2 wassail-eval
```

Once the Docker image has been loaded, you can run the following commands to
obtain a shell in the appropriate environment:
```
docker volume create result
docker run -it -v result:/tmp/out/ wassail-eval bash
su - opam
```

# Reproducing results of RQ1
Our manual translations of the "classical" examples are included in the `rq1/`
directory (available in the docker image and in `rq1.tar.xz`). We
include the slices computed by our implementation in the `rq1/out/` directory.

A slice can be produced for each example in the docker image as follows, where
the first argument is the name of the program being sliced, the second the
function index being sliced, the third the slicing criterion (indicated as the
instruction index, where instructions start at 1), and the last argument is the
output file for the slice:

```
cd rq1/
wassail slice scam-mug.wat 5 8 scam-mug-slice.wat
wassail slice montreal-boat.wat 5 19 montreal-boat-slice.wat
wassail slice word-count.wat 1 41 word-count-slice1.wat
wassail slice word-count.wat 1 43 word-count-slice2.wat
wassail slice word-count.wat 1 39 word-count-slice3.wat
wassail slice word-count.wat 1 45 word-count-slice4.wat
wassail slice word-count.wat 1 37 word-count-slice5.wat
wassail slice agrawal-fig-3.wat 3 38 agrawal-fig-3-slice.wat
wassail slice agrawal-fig-5.wat 3 37 agrawal-fig-5-slice.wat
```

The slice results can then be inspected manually, and compared with the original
version of the .wat program to see which instructions have been removed, or with
the expected solutions in the `out/` directory, e.g. by running:
```
diff word-count-slice1.wat out/word-count-slice1.wat
```
(No output is expected if the slice is correct)

# Reproducing results of RQ2, RQ3, and RQ4
For these RQ, we include the data resulting from our evaluation, but we also
allow reviewers to rerun the full evaluation if needed. However, such an
evaluation requires a heavy machine and takes quite some time (4-5 days to run
to completion with a 4 hours timeout). In our case, we used a machine with 256
GB of RAM and a 64-core processor with HyperThreading enabled, allowing us to
run 128 slicing jobs in parallel.

## Runnig the Evaluation
We explain how to run the full evaluation, or only a partial evaluation below.
One can directly skip to the next section and reuse our raw evaluation results,
provided alongside this artifact.

### Running the Full Evaluation
In order to reproduce our evaluation, you can run the following commands in the
docker image. It is recommended to run them in a tmux session if one wants to
inspect other elements in parallel (tmux is installed in the docker image). The
timeout (set to 4 hours per binary, like in the paper) can be decreased by
editing the `evaluate.sh` script (vim is installed in the docker image).

This is expected to take 2-3 days of time, on a machine with 128 cores.
In order to produce only partial results, see the next section.

```
cd filtered
cat ../supported.txt | parallel --bar -j 128 sh ../evaluate.sh {}
```

The results are outputted in the `/tmp/out/` directory.

### Running a Partial Evaluation
If one does not have access to a high-end machine with 128 cores nor the time to
run the full evaluation, it is possible to produce partial results. To do so,
the following commands can be run. This will run the evaluation on the full
dataset in a random order, which can be stopped early to represent a partial
view of our evaluation, on a random subset of the data. In order to gather more
datapoints, it is also advised to decrease the timeout in the `evaluate.sh`
file, for example to 20 minutes by setting `TIMEOUT=20m` with `nano
evaluate.sh`. The number of slicing jobs running in parallel can also be
decreased to match the number of processors on the machine running the
experiments (the `-j 128` argument in the following command runs 128 parallel
jobs)

```
sudo chown opam:opam /tmp/out/
cd filtered
shuf ../supported.txt | parallel --bar -j 128 sh ../evaluate.sh {}
```

The evaluation results will be stored in the `/tmp/out/` directory.

### Skipping the Evaluation Run
Instead of rerunning the evaluation, one can rely on our full results included
in the `data.txt.xz` and `error.txt.xz` archives. These can simply be downloaded
from within the Docker machine and extracted in `/tmp/out/`:

```
cd /tmp/out/
wget https://zenodo.org/record/5821007/files/data.txt.xz
wget https://zenodo.org/record/5821007/files/error.txt.xz
unxz data.txt.7z
unxz error.txt.7z
```

## Processing the data

In order to process this data, we included multiple python script.
These require around 100GB of RAM to load the full dataset in memory.
The scripts should be run with Python 3.
When running this in the docker image, first run `cd /tmp/out/ && cp /home/opam/*.py ./`
- To count the number of functions sliced, run `cut -d, -f 1,2 data.txt | sort
  -u | wc -l`. This takes around 6 minutes to run on the full dataset.
- To count the total number of slices encountered, run `wc -l data.txt
  error.txt`. This takes around 15 seconds to run.
- To count the number of errors encountered, run `wc -l error.txt`. This takes
  around 1 second to run.
- To produce data and graphs regarding the sizes and timing, run `python3
  statistics-and-plots.py`. This will output the statistics presented in the
  paper, along with Figure 2 (rq2-sizes.pdf) and Figure 3 (rq2-times.pdf). This
  script takes around 35 minutes to run.
- To find the executable slices that are larger than the original programs, run
  `python3 larger-slices.py > larger.txt`. This script takes around 2h30 to
  run. It will list the slice using the notation `filename function-sliced
  slicing-criterion` in the larger.txt file, from which the slice can be
  recomputed by running `wassail slice function-sliced slicing-criterion
  output.wat` in the docker image. It will also output statistics regarding
  these slices, which you can easily inspect by running `tail larger.txt`.
- To investigate slices that could not be computed, run:
  ```
  sed -i error.txt -e 's/annotation,/annotation./'
  python3 errors.py
  ```
  This will take a few seconds to run and will print a summary of the errors
  encountered during the slicing process, and requires some manual sorting to map
  to the categories we discuss in the paper. Here is a summary of the errors
  encountered and their root cause:

### Root Cause: Unsupported Usage of br_table
Error: (Failure"Invalid vstack when popping 2 values")
Error: (Failure"Spec_inference.drop: not enough elements in stack")
Error: (Failure"Spec_inference.take: not enough element in var list")
Error: (Failure"unsupported in spec_inference: incompatible stack lengths (probably due to mismatches in br_table branches)")
### Root Cause: Unreachable Code
Error: (Failure"Unsupported in slicing: cannot find an instruction. It probably is part of unreachable code.")
Error: (Failure"bottom annotation")
Error: (Failure"bottom annotation. this an unreachable instruction")

# RQ5: Comparison to Slicing C Programs
For this RQ, we include the following data in the `rq5.7z` archive, and in the `rq5/` directory in the docker image:
- The slicing subjects in their C and textual wasm form in `rq5/subjects/`
- The CodeSurfer slices in their C and textual wasm form in `rq5/codesurfer/`
- Our slices in their wasm form in `rq5/wasm-slices/`

As this RQ requires heavy manual comparison, we do not expect the reviewers to
reproduce all of our results. We include a summary of our manual investigation
in `rq5-manual.txt`. In order to validate these manual findings, one can for
example inspect a specific slice. For example, the following line in
`rq5-manual.txt`:

```
adpcm_apl1_565_expr.c.wat INTERPROCEDURAL
```

can be validated as follows:
```
cd ~/
# This generates a trimmed down version of the CodeSurfer slice, only containing the function of interest
wassail count-in-slice rq5/codesurfer/adpcm_slices/adpcm_apl1_565_expr.c.wat slice.wat
# This compares the CodeSurfer slice with our slice
diff --side-by-side slice.wat rq5/adpcm_apl1_565_expr.c.wat
```

In this case, most extraneous instructions are present in the CodeSurfer slices,
at the end of the function. This indicates that these are present in order to
preserve interprocedural behavior, which corresponds to the `INTERPROCEDURAL`
tag in the `rq5-manual.txt`