Reproduction Package for ESEC/FSE 2020 Article `Domain-Independent Interprocedural Program Analysis using Block-Abstraction Memoization'

This file describes the contents of the Reproduction package for the article
"Domain-Independent Interprocedural Program Analysis using Block-Abstraction Memoization",
Proc. ESEC/FSE 2020. ACM.

# License:

SPDX-License-Identifier: CC-BY-4.0
https://spdx.org/licenses/CC-BY-4.0.html

Some parts of this archive are licensed differently.
For details see `LICENSE.txt`.

# Contents of this Archive:

- `LICENSE.txt` :         specifies the license
- `README.txt` :         this file
- `benchmarks/` :        benchmark definition(s) for different configurations of CPAchecker
- `results/` :           results of the execution of CPAchecker with different domains on recursive  tasks
  - `*.xml.bz2` :        XML results from BenchExec
  - `*.logfiles.zip` :   output of CPAchecker
- `results-verified/` :  results of verification runs for recursive tasks,
                    extracted from the results archive of SV-COMP 2020 (https://doi.org/10.5281/zenodo.3630205)
- `scripts/` :           utility scripts to execute benchmarks and generate tables
- `sv-benchmarks/`    :  verification tasks extracted from the sv-benchmark repository
                    used in SV-COMP 2020 (https://zenodo.org/record/3633334)
- `tables/` :            generated HTML tables and a LaTeX file for the user
- `tools/` :             tools BenchExec and CPAchecker to evaluate the results and generate tables

All benchmarks were executed based on the following components:
- git@github.com:sosy-lab/sv-benchmarks.git tag:svcomp20
- git@github.com:sosy-lab/benchexec.git 2.5.1
- https://cpachecker.sosy-lab.org/CPAchecker-1.9-unix.tar.bz2

The remaining part of this README is structured as follows:
first we describe the single-task execution of CPAchecker,
then we provide some steps towards executing a whole benchmark for CPAchecker and multiple tasks.
The last section describes the scripts that can be used for an automated re-evaluation of the benchmarks
that were used for the paper.

# Execute CPAchecker on a Single Verification Task

CPAchecker runs best on a recent Ubuntu 18.04 (x86_64) and
requires Java 11 (or newer), which is not part of this archive.

For the execution of a single verification task, switch the current working directory:

    cd tools/CPAchecker-1.9-unix/

There are different configurations available for evaluation.
The following configurations correspond to the approaches explained in the article:
- valueAnalysis-bam-rec
- predicateAnalysis-bam-rec
- intervalAnalysis-bam-rec
- svcomp20

Note that 'svcomp20' applies strategy selection, i.e.,
for non-recursive verification tasks it might choose another analysis.

Choose your preferred domain-specific configuration, then execute CPAchecker:

    scripts/cpa.sh -timelimit 900s -stats \
        -predicateAnalysis-bam-rec \
        -spec ../../sv-benchmarks/c/properties/unreach-call.prp \
        ../../sv-benchmarks/c/recursive-simple/sum_non_eq-2.c

This command with the specified task will execute CEGAR and iterate twice.
The first iteration will unroll the recursion once before finding a spurious counterexample.
The second iteration will unroll the recursive procedure twice before proving safety.
CPAchecker will then report `Verification result: TRUE. No property violation found`.
This command will produce several files for the user in the directory 'output/',
including (GraphViz) dot-files for the control-flow automata and abstract reachability graphs,
an interactive HTML-based report, and several log files with additional data.

# Execute CPAchecker for Several Verification Tasks of a Benchmark

This section is for more advanced users of CPAchecker!
Executing several tasks via BenchExec allows precise measurements, resource limitations,
and the generation of tables afterwards.

For the execution of a benchmark, switch the current working directory:

    cd tools/CPAchecker-1.9-unix/

Depending on the system environment, please configure CGroups for BenchExec.
On a default Ubuntu 18.04 with a local filesystem and system environment,
executing the following steps should be sufficient:

    sudo swapoff -a
    sudo chmod o+wt '/sys/fs/cgroup/cpuset/'
    sudo chmod o+wt '/sys/fs/cgroup/cpu,cpuacct/user.slice'
    # or:  sudo chmod o+wt '/sys/fs/cgroup/cpu,cpuacct/'
    sudo chmod o+wt '/sys/fs/cgroup/freezer/'
    sudo chmod o+wt '/sys/fs/cgroup/memory/user.slice'
    # or: sudo chmod o+wt '/sys/fs/cgroup/memory/'

During the benchmarking process, please read the information and hints printed by BenchExec carefully.
In case of errors or warnings, please consider looking at the BenchExec documentation at
'https://github.com/sosy-lab/benchexec'.

To perform a benchmark, execute BenchExec locally:

    ../benchexec/bin/benchexec ../../benchmarks/cpachecker.xml -o ../../results/ -t ReachSafety-Recursive

Add the option '-c 8', to specify the number of CPU cores (default is 8).
Add the option '-T 900s', to specify the time limit (default is 900s).
Add the option '-M 15GB', to specify the memory limit (default is 15GB).
Please note that not all configurations are intended for usage with less ressources,
e.g., CPAchecker's configuration 'svcomp20' is optimized for 900 s runtime.
The option '-t ReachSafety-Recursive' limits the execution to the set of recursive verification tasks only.
We provide a wider range of programs and results in the archive.
Please note that our approach was mainly developed to handle recursive verification tasks
and might not perform well when analyzing arbitrary programs.

The benchmark results will be available in the directory '../../results/' afterwards.

# Replicate the Experiments in the Article

The scripts in the directory 'scripts/' are intended to exactly execute the experiments reported in the article,
to be executed from the root directory of this archive:

    scripts/executeBenchmark.sh : execute CPAchecker on several verification tasks
    scripts/generateTables.sh   : convert the results of the evaluation into HTML tables

Please take a look in the description above for configuring CGroups for BenchExec
before starting the benchmarking process, since the experiments require CGroups.
Please also clean the directory 'results/' before running a new experiment or creating any new tables!
On a machine with less than 15 GB of memory or less than 8 CPU cores,
the options for the benchmark execution should be adjusted, which is also described below.
For most verification tasks, a time limit of 20 s and a memory limit of 2 GB is sufficient.
Depending on the machine, the evaluation of only the category ReachSafety-Recursive alone requires about 16 h of runtime.
Further categories require more time. Please take a look at the provided tables for the estimated runtime.

The results from SV-COMP 2020 were taken directly from the archive (https://doi.org/10.5281/zenodo.3630205).
For an evaluation of tools participating in SV-COMP, we refer the reader the archive,
or to the competition web site https://sv-comp.sosy-lab.org/2020.

We provide a LaTeX document `tables/table-tools.tex`
which references automatically generated files like `tables/data.tex`.
This document can be used to generate a PDF document `tables/table-tools.pdf`
with the tables as provided in the article.
Please note that measurements of CPU time and memory can have a small variance,
even if we use BenchExec to be as precise as possible.