DarthShader: Fuzzing WebGPU Shader Translators & Compilers

# DarthShader

DarthShader is an academic prototype of a fuzzer that aims at fuzzing the WebGPU shader compilation pipeline in web browsers. To this end, the fuzzer uses a combination of code generation and mutation to produce WebGPU shaders (wgsl). These wgsl shaders are then processed by a shader translator (the front-end). In particular, we support the shader translators of Chrome (tint), Firefox (naga), and Safari (wgslc). These components translate the input shader into a representation native to the respective GPU stack of the host operating system. For example, on Windows the WebGPU shaders are translated to HLSL. Once this native representation has been generated, shaders are then further processed by the respective graphics stack (the back-end). On Windows, HLSL shaders are consumed by dxcompiler, the shader compiler of DirectX12. DarthShader also supports testing of this component: First, WebGPU shaders are translated with tint to HLSL. Next, the HLSL output is fed into dxcompiler. This two-stage approach replicates the shader translation pipeline and hence ensures that crashes in dxcompiler are reachable from the web. 

## Note on Artifact Availability
Towards the end of the artifact evaluation period, we will:
1. Upload DarthShader, the provided harnesses and documentation on GitHub
2. Upload the provided zip file to Zenodo
3. Upload a prebuilt docker image to Zenodo

## Quickstart
```bash
# Build the docker image which contains:
# DarthShader (and its competitors), seeds, and the targets.
# This build process might take some time and requires an active
# internet connection for pulling multiple repos
sudo docker build -t darthshaderae buildfiles

# disable core dumps
sudo sysctl -w 'kernel.core_pattern=|/bin/false'

# set CPU governor to performance
echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

# Run the coverage experiment from the paper.
# This will also generate results for unique coverage and 
# semantic correctness rate. The results are stored in evalresult
sudo docker run -it -v "$(pwd)/"evaldata:/artifact/eval/data -v "$(pwd)/"evalresults:/artifact/eval/results darthshaderae /usr/bin/python3 artifact/eval/evaluate.py coverage

# Run the ablation study from the paper.
# The ablation coverage plots are stored in the evalresults folder
sudo docker run -it -v "$(pwd)/"evaldata:/artifact/eval/data -v "$(pwd)/"evalresults:/artifact/eval/results darthshaderae /usr/bin/python3 artifact/eval/evaluate.py ablation
```

## Docker Image Layout
| Container Folder         | Content |
|:-------------------------|:--------|
| `/artifact/fuzzer`       | Fuzzer executables for DarthShader and its competitors |
| `/artifact/ablation`     | Ablations of DarthShaders for the ablation experiment  |
| `/artifact/target`       | Fuzzing targets instrumented with AFL++                |
| `/artifact/coverage`     | Fuzzing targets instrumented for lcov coverage         |
| `/artifact/seeds`        | WGSL seed files extracted from the dawn repo           |
| `/artifact/eval`         | Evaluation script and helper executable                |
| `/artifact/eval/data`    | Bind-mounted folder with fuzzing queues and cov data   |   
| `/artifact/eval/results` | Bind-mounted folder with plots, eval results, and logs |

## Troubleshooting
In case of any issues during the artifact evaluation, please provide the LOGS, WARNINGS and ERRORS files stored in the `evalresults` directory. Note that these files might be absent if no logs at the respective log-level exist.

## DarthShader Command Line Arguments

The usage of DarthShader is best explained by its `--help` switch:

```console
Usage: darthshader [OPTIONS] --output <out> <exec> [-- [arguments]...]

Arguments:
  <exec>          The instrumented binary we want to fuzz
  [arguments]...

Options:
  -o, --output <out>              The directory to place finds in ('corpus')
  -i, --input <in>                The directory to read initial inputs from ('seeds')
  -l, --logfile <logfile>         Duplicates all output to this file [default: libafl.log]
  -t, --timeout <timeout>         Timeout for each individual execution, in milliseconds [default: 1200]
  -d, --debug-child               If not set, the child's stdout and stderror will be redirected to /dev/null
      --iterations <iterations>   Number of iterations to fuzz
      --config <generatorconfig>  The .yaml generator config file
      --seed <generatorseed>      Seed for the generation process
      --discard-crashes           Do not store crashes but explore for coverage only
  -h, --help                      Print help
  -V, --version                   Print version
```

The most important arguments are:
- The `exec` argument specifies the fuzzing harness executable.
- The `--output` folder contains two sub-directories. There is the `queue` file which contains all samples contributing new coverage. In addition, there is a `crashes` directory containing, as the name implies, all crashes found during the fuzzing campaign. Note that the samples are serialized in `.ron` files. Converting them to `.wgsl` files is possible using the lifter. This utility program is stored in the docker image at `/artifact/eval/darthshader_lifter`.
- The `--input` folder will be traversed recursively for all shader files for importing them.

## Further Components

### ****Seeds****
The fuzzer supports importing seeds from in the following formats: wgsl, spv, and glsl. In our evaluation (and as a good starting point), we use the [`dawn`](https://dawn.googlesource.com/dawn) repository as source, as it contains thousands of shaders.

### ****Compiler pass****
The fuzzing targets are instrumented with a variant of the well-known AFL++ instrumentation. More precisely, the only difference is that instead of collecting hit-counts, we only collect information about whether an edge was hit (or not). Please note that the AFL++ forkserver was recently updated. The libAFL version used by DarthShader is incompatible with these changes. Hence building with an older version of AFL++ (e.g., 4.10c) is required.

### ****Targets****
DarthShader currently provided fuzzing harnesses for 4 different shader compilers. These are 3 front-end translators used directly in browsers (tint, naga, wgslc) and the back-end compiler of DirectX, dxcompiler. For each of the fuzzing targets, the repository contains patches that add an AFL forkserver to the target.

#### ****tint****
This is the shader compiler of Chrome, supporting compilation from WGSL to the HLSL (Windows), SPIR-V (Linux & Android), and Metal (OSX). Our fuzzing harness for tint translates a single WGSL shader to each of the three target languages.

#### ****dxcompiler****
This is the DirectX shader compiler taking HLSL as input and producing an output format based on LLVM IR. Our fuzzing harness first translates WGSL shaders to HLSL via tint and subsequently passes the HLSL code to dxc, our setup hence replicates browser usage.

#### ****naga****
This is the shader compiler of Firefox supporting compilation from WGSL to HLSL, SPIR-V, and Metal. Analogue to the tint harness, our naga harness translates a single WGSL shader to HLSL, SPIR-V, and Metal.

#### ****wgslc****
The shader compiler of Safari translating WGSL to Metal. No other output languages are supported.

## Artifact Evaluation Experiments

### Coverage, Unique Coverage, Correctness
The data necessary for computing the correctness rate (Table 1), coverage over time (Figure 4 + Figure 8a), and unique coverage (Figure 5) is all collected at the same time. Hence, running the `coverage` experiment will also produce the evaluation results for unique coverage and correctness rate. After building the container as described above, run the experiment as follows:

```bash
# Run the coverage experiment from the paper.
# This will also generate results for unique coverage and
# semantic correctness rate. The results are stored in evalresult
sudo docker run -it -v "$(pwd)/"evaldata:/artifact/eval/data -v "$(pwd)/"evalresults:/artifact/eval/results darthshaderae /usr/bin/python3 artifact/eval/evaluate.py coverage
```

The results of this evaluation are stored in the `evalresults` folder. There are coverage plots for each of the 4 evaluation targets (e.g., `coverage_dxc.svg`). These coverage plots should match the ones in Figure 4 and Figure 8a. Note that due to inherent randomness of the fuzzing process and other factors, such as differences in clock speed, the plots are not expected to be identical but should be similar.
There are five unique coverage plots stored in the same folder. There is one for naga, wgslc, and dxc (e.g., stored in `evalresults/unique_coverage_dxc_uninformed.svg`). For tint, there are two separate plots, one with fuzzers having access to seeds and one plot for the fuzzers without seeds. The plots should correspond to Figure 5 in the paper.
Finally, this experiment collects correctness rates for all the fuzzers on each front-end translator (tint, naga, and wgslc). These results are stored in `evalresults/correctness`

### Ablation Study
The ablation study measures the impact of individual mutation strategies. In the paper, this corresponds to Figure 6 and Figure 8b. We either disable or delay AST and IR mutations. In total, the runtime of each campaign is 48h. Depending on the ablation, IR/AST mutations remain disabled over the entire campaign or are enabled after 24h. After building the docker image as explained above, the ablation experiment can be started as follows:

```bash
# Run the ablation study from the paper.
# The ablation coverage plots are stored in the evalresults folder
sudo docker run -it -v "$(pwd)/"evaldata:/artifact/eval/data -v "$(pwd)/"evalresults:/artifact/eval/results darthshaderae /usr/bin/python3 artifact/eval/evaluate.py ablation
```

The results of the ablation study are stored in the `evalresults` folder. There are coverage plots for each of the four evaluation targets (e.g., `ablation_dxc.svg`). Again, the coverage plots are not necessarily identical, but the general effect should be clearly visible.