Artifact for " Fast Shadow Execution for Debugging Numerical Errors using Error Free Transformations"

# Artifact for the paper "Fast Shadow Execution for Debugging Numerical Errors using Error Free Transformations" at OOPSLA 2022

## Abstract of the paper

This paper proposes, EFTSanitizer, a fast shadow execution framework for detecting and debugging numerical errors during late stages of testing especially for long-running applications. Any shadow execution framework needs an oracle to compare against the floating point (FP) execution. This paper makes a case for using error free transformations, which is a sequence of operations to compute the error of a primitive operation with existing hardware supported FP operations, as an oracle for shadow execution. Although the error of a single correctly rounded FP operation is bounded, the accumulation of errors across operations can result in exceptions, slow convergences, and even crashes. To ease the job of debugging such errors, EFTSanitizer provides a directed acyclic graph (DAG) that highlights the propagation of errors, which results in exceptions or crashes. Unlike prior work, DAGs produced by EFTSanitizer’s include operations that span various function calls while keeping the memory usage bounded. To enable the use of such shadow execution tools with long-running applications, EFTSanitizer also supports starting the shadow execution at an arbitrary point in the dynamic execution, which we call selective shadow execution. EFTSanitizer is an order of magnitude faster than prior state-of-art shadow execution tools such as FPSanitizer and Herbgrind. We have discovered new numerical errors and debugged them in many long-running applications using EFTSanitizer.

## Claims to validate/reproduce

In this artifact, we provide instructions to reproduce and validate the following claims in the paper.

1. Ability to detect known errors in microbenchmarks.

2. Debugging with gdb to diagnose the bug in the case study.

3. Performance slowdown of EFTSanitizer

4. Speedups for EFTSanitizer when compared to FPSanitizer

Note: To make it feasible to run the artifact quickly, we have created mini-versions especially for (4) above. Otherwise it takes more than a day on a native machine to get the performance numbers.

## Instructions to run the artifact

### Downloading prebuilt Docker Image

1. Install docker if it is not installed already by following the documentation at https://docs.docker.com/install/.

2 Download the attached docker image.

3. Run the docker image with the following commands:

docker load < eftsanitizer-final-submit.tar.gz

docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -it eftsanitizer-final-submit

You may be required to run as sudo if you are using Docker on Linux.

4. You should get a shell running the container.

5. Set the environment variables in the container by executing the following commands:

export FPSAN_HOME=/root/artifact_build/fpsanitizer

cd /root source mypath.sh

## Ability to detect known errors in microbenchmarks

In this experiment, you will run the correctness suite provided with EFTSanitizer to test the ability to detect errors. There are 46 microbenchmarks in total.

Run them with the provided scripts as follows:

cd /root/artifact_build/EFTSanitizer/correctness_test

python3 correctness_test.py

You will observe the following output:

Running test: newtonraphsonsqrt

Running test: SinTaylorExpansion

Running test: gauss_pp

Running test: diff-simple

Running test: newtonraphson

Running test: simpl-swallow-testc

Running test: simpl-swallow-testa

Running test: newton

Running test: indirect

Running test: indirect1

Running test: simpl-swallow-testd

Running test: SimpsonsRule

Running test: cc

Running test: same-small

Running test: naturalnumber

Running test: rootfinding2

Running test: muller

Running test: rootfinding1

Running test: cordic_cos

Running test: var-test-2

Running test: simpl-swallow-testb

Running test: mini

Running test: dotproduct2

Running test: second_order

Running test: small

Running test: var-test

Running test: multi-args

Running test: MindlessG

Running test: redir

Running test: ParallelResistance

Running test: gromacs-bondfree-580

Running test: diff-roots-simple

Running test: sum-500

Running test: tiny

Running test: gsl-modpi

Running test: exact-ranges

Running test: sum-50

Running test: ContinuedFraction1

Running test: compare-zero-self

Running test: sqrtf

Running test: dotproduct1

Running test: diff-roots

Running test: simple-negate

Running test: cordic_sin

Running test: onthefly

Running test: some-trig

Total number of benchmarks: 46(expected)

Number of benchmarks with high rounding error: 23(expected)

Number of benchmarks with NaN computation: 0(expected)

Number of benchmarks with Inf computation: 2(expected)

Number of benchmarks with branch flip: 5(expected)

Number of benchmarks with integer conversion error: 0(expected)

## See the attached pdf for reproducing performance overhead graphs