Reproduction Package (Reduced Version) for Article "Decomposing Software Verification into Off-the-Shelf Components: An Application to CEGAR"

[Attention: This is a reduced version of the supplementary artifact, for convenient download. It does not contain the sv-benchmarks.zip and intermediate files generated during C-CEGAR execution. See the next version for the full supplementary artifact: https://doi.org/10.5281/zenodo.5443638.]

Decomposing Software Verification into Off-the-Shelf Components: An Application to CEGAR

Abstract: Tools for software verification are typically cohesive units of software with tightly coupled components. This makes it difficult to re-use components, and the potential for workload distribution is limited. Innovations in software verification might find their way into practice faster if provided in smaller, more specialized components.

In this paper, we propose to strictly decompose software verification: the verification task is split into independent subtasks, implemented by only loosely coupled components communicating via clearly defined interfaces. We apply this decomposition concept to one of the most frequently employed techniques in software verification: counterexample-guided abstraction refinement (CEGAR). CEGAR is a technique to iteratively compute an abstract model of the system. We develop a decomposition of CEGAR into independent components with clearly defined interfaces that are based on existing, standardized exchange formats.

We implement this component-based CEGAR (C-CEGAR) to investigate the feasibility of decomposition for verification. The decomposition concerns the three core tasks of CEGAR: abstract-model exploration, feasibility check, and precision refinement. We experimentally show that — despite the necessity of exchanging complex data via interfaces — the efficiency thereby only reduces by a small constant factor while the precision in solving verification tasks even increases. We furthermore illustrate the advantages of C-CEGAR by experimenting with different implementations of components, thereby further increasing the overall effectiveness and testing that substitution of components works well.

Contents

This document supports the following claims of our submission:

1. The boolean expression 1 <= y + 2 ((y * -1 + 1) / 2)* is equivalent to (y mod 2 = 1)
2. Ultimate Automizer is the only reasonable choice for invariant generator, next to CPAchecker.
3. All data and experimental results mentioned.

In addition, all used tools are provided as archives in directory used-tools/, and the benchmark tasks used are provided in sv-benchmarks.zip.

The changes done to the wrapper script of Ultimate Automizer (as described in Sect. 5.2) are listed in ultimate-automizer-wrapper-changes.diff.

Equivalence of Boolean Expressions

See reasoning-predicates.pdf for a reasoning about the equivalence of the boolean expressions 1 <= y + 2 ((y * -1 + 1) / 2)* and (y mod 2 = 1) (according to C language semantics).

Choice of Invariant Generators

Page overview-verifiers.html gives an overview of all considered verifiers and our selection criteria.

Experiment Data

Raw Data

All experimental raw data can be found in directory raw-data/. Page overview-data.table.html gives an overview on all benchmark results (also available as CSV).

Reproducing Figures

To reproduce all figures of the paper, use the prepared files available in directory figures/.

This requires Python 3.8 and the following python modules:

• numpy
• matplotlib
• scipy

You can install these modules with pip install numpy matplotlib scipy.

Python script create_figures.py produces the plots and figures of our submission from the CSV files iterations.table.csv (comparison between Pred and C-Pred) and rq3.csv (comparison between the used C-PredWit configurations).

To reproduce all files, run on the command line:

> cd figures
> python3 create_figures.py

As a result, multiple files should appear in directory figures/:

• boxplot.pdf: Figure 10b
• boxplot-with-outliers.pdf: the boxplot of Figure 10b, but outliers are included.
• rq3.pdf: Figure 13
• runtime-to-it-median.pdf: Figure 11
• runtime-to-it-median-with-outliers.pdf: Figure 11, but outliers are included
• scatter.pred-vs-mpred.pdf: Figure 10a