Reproduction Package for TACAS 2021 Submission `Interpolation and SAT-Based Model Checking Revisited'

# Reproduction Package

## Interpolation and SAT-Based Model Checking Revisited: Adoption to Software Verification

## Abstract

This artifact is a reproduction package for the paper
"Interpolation and SAT-Based Model Checking Revisited: Adoption to Software Verification"
(submitted to TACAS'21).

It contains code and data used in the evaluation of the paper,
including CPAchecker with the implementation described in the paper,
the benchmark suite used for the evaluation,
the raw data resulting from the evaluation,
scripts to run the experiments,
and instructions for reproducing the tables and figures in the paper.

A full reproduction of the evaluation needs 16 GB of memory and about half a year of CPU time,
but a limited subset of the benchmark tasks can be used for demonstration purposes
if at least two CPU cores and 4 GB of memory are available.

This artifact is published at https://doi.org/10.5281/zenodo.4090419 (DOI 10.5281/zenodo.4090419).

## Content

This artifact contains the following items:

- `Readme.*`: this documentation
- `License.txt`: license information of the artifact
- `Interpolation_and_SAT-Based_Model_Checking_Revisited.pdf`: a preprint of the paper
- `cpachecker`: a directory containing [CPAchecker](https://cpachecker.sosy-lab.org)
  at revision [`cfa-single-loop-transformation:34785`](https://github.com/sosy-lab/cpachecker/commit/760c503f9f7523e7515972286bf2a669a5c24729) (sources plus binary),
  which was used in our evaluation
- `sv-benchmarks`: a directory containing the benchmark suite used in [SV-COMP'20](https://sv-comp.sosy-lab.org/2020/) and our evaluation
  (subset of tag `svcomp20` of [SV-Benchmarks](https://github.com/sosy-lab/sv-benchmarks/tree/svcomp20))
- `packages`: the necessary Ubuntu packages to run the experiments on the [TACAS'21 AE VM](https://doi.org/10.5281/zenodo.4041464)
- `data-submission`: raw data produced from our experiments, both from the full experiment as well as a small demo subset
- `TACAS21-interpolation-model-checking.xml`: the benchmark definition of the experiments for [BenchExec](https://github.com/sosy-lab/benchexec)
- `Makefile`: a file to execute the commands

This readme file will guide you through the following steps:

1. Setup of CPAchecker and BenchExec
2. Running the example verification task used in the paper
3. Performing experiments over a selected category and the complete benchmark suite
4. Analyzing the experimental data
5. Discussing known issues of the artifact

### TL;DR

Type the following commands to:

- Install Java 11: `make install-java-11`
- Install BenchExec:
  - `make install-benchexec`
  - reboot the system
  - `make prepare-benchexec` (to turn off swap memory and allow user namespaces, needs to be redone after every reboot)
  - `make test-cgroup` (to test if cgroup is configured correctly)
- Test if CPAchecker can be run normally: `make test-cpachecker`
- Test if BenchExec is installed successfully: `make test-benchexec`
- Run an example for interpolation-based model checking: `make example-imc`
- Run an example for predicate abstraction: `make example-pred`
- Run an example for k-induction: `make example-kind`
- Perform a demo experiment on subset of tasks: `make demo-run`  
  In order to require less resources we recommend `make timelimit=60s memlimit=3GB demo-run` to set the timelimit to `60s` and memlimit to `3GB` per task.
  Note that this still requires to increase the RAM of the TACAS'21 AE VM.
- Perform a complete experiment: `make full-run` (needs 16 GB of RAM and 168 days)
- View an HTML file of the submitted results from a demo run: `make submitted-demo-table`
- View an HTML file of the submitted results from a full run (the data used in the paper): `make submitted-full-table`

For the full description, see the Readme files.