Reproduction Package for ASE 2023 Article `CPA-DF: A Tool for Configurable Interval Analysis to Boost Program Verification'

Reproduction Package

CPA-DF: A Tool for Configurable Interval Analysis to Boost Program Verification

 

Abstract

This artifact is a reproduction package for the paper “CPA-DF: A Tool for Configurable Interval Analysis to Boost Program Verification”, published at ASE 2023 Tool Demonstrations Track.

It consists of source code, precompiled binaries, and input data used in the evaluation of the paper, as well as the results produced from the experiments. Specifically, it includes the source code of CPA-DF, the SV-COMP ’23 benchmark suite, the binaries of the software verifiers used in the evaluation, the experimental data generated from the evaluation, and instructions to run the tools and experiments.

This reproduction package works best with the SoSy-Lab Virtual Machine, which runs Ubuntu 22.04 LTS. (If you test this artifact with the SoSy-Lab VM, you can skip all installation steps below.)

Demonstration video: https://youtu.be/l7UG-vhTL_4

Contents

This artifact contains the following items:

• README.html: this documentation
• License.txt: license information of the artifact
• example.c: an example C program for demonstration
• cpachecker/: a directory containing the source code and precompiled binaries of CPA-DF
• coveriteam/: a directory containing the source code of CoVeriTeam
  • coveriteam/cache/: a directory containing the binaries of the evaluated software verifiers
  • cvt-files/: a directory containing programs written in the CoVeriTeam language and actor definitions
• sv-benchmarks/: a directory containing the SV-COMP ’23 benchmark tasks used in our evaluation
• paper-results/: a directory containing the experimental results produced from our full evaluation
• bench-defs/: a directory containing the benchmark definitions of the experiments (used by BenchExec, a framework for reliable benchmarking)

This readme file will guide you through the following steps:

• Execute CPA-DF and other analyses in CPAchecker
• Execute verifiers and portfolios via CoVeriTeam
• View the experimental results in the paper
• Reproduce the full experiments

Execute CPA-DF and other analyses in CPAchecker

System requirements for CPAchecker

To execute CPAchecker, Java Runtime Environment (JRE) version ≥ 17 is required. For more information, please refer to cpachecker/INSTALL.html.

Run verification algorithms in CPAchecker

Below we list the commands to run the evaluated verification algorithms in this paper. These example commands use example.c as the input program. You can replace it with another program.

• CPA-DF with dynamic precision refinement

  # expected verification result: TRUE
  cpachecker/scripts/cpa.sh -32 -spec sv-comp-reachability -dataFlowAnalysis example.c

• CPA-DF with static precision (16,t)

  # expected verification result: UNKNOWN
  cpachecker/scripts/cpa.sh -32 -spec sv-comp-reachability -invariantGeneration -setprop cpa.invariants.interestingVariableLimit=16 example.c

  • To adjust the number of important variables, change the parameter cpa.invariants.interestingVariableLimit
  • To disable widening, add -setprop cpa.invariants.abstractionStateFactory=NEVER to the command (the expected verification result becomes TRUE if widening is disabled)
  • The configuration is named -invariantGeneration because it originally worked as an invariant generator for k-induction in CPAchecker (see the CAV 2015 paper for more detail).

• Plain k-induction (KI)

  # expected verification result: UNKNOWN
  cpachecker/scripts/cpa.sh -32 -spec sv-comp-reachability -config cpachecker/config/components/kInduction/kInduction.properties example.c

  • Note that KI alone is not able to verify the example program within the time limit (15 min)

• Parallel portfolio of k-induction and CPA-DF (KI || DF, with the time limit of CPA-DF set to 5 min)

  # expected verification result: TRUE
  cpachecker/scripts/cpa.sh -32 -spec sv-comp-reachability -kInduction-dfInvariants300s-noCoop example.c

• Injecting invariants produced by CPA-DF to k-induction (KI <- DF)

  # expected verification result: TRUE
  cpachecker/scripts/cpa.sh -32 -spec sv-comp-reachability -kInduction-dfInvariants300s example.c

For more information on how to run CPAchecker, see cpachcker/README.html.

Understand the results

The following is an example output shown on the console after an analysis is finished.

[...redacted...]

Verification result: TRUE. No property violation found by chosen configuration.
More details about the verification run can be found in the directory "./output".
Graphical representation included in the file "./output/Report.html".

There are 3 possible outcomes of the verification result:

1. TRUE: the program is “safe”, i.e., it satisfies the given safety property
2. FALSE: the program is “unsafe”, i.e., it contains a violation to the given safety property
3. UNKNOWN: the analysis was inconclusive

CPAchecker also generates an HTML report for conveniently browsing the results. Please refer to cpachecker/doc/Report.html for more information.

Execute verifiers and portfolios via CoVeriTeam

System requirements for CoVeriTeam

CoVeriTeam requires Python version ≥ 3.7 and a Linux-based operating system with Cgroups v1. For more information, please refer to coveriteam/README.html. In addition, the system has to satisfy the requirements of the executed verifiers.

Run a single verifier

The following command executes Symbiotic via CoVeriteam and uses example.c as the input program.

# to run another verifier, replace "symbiotic" with "esbmc" or "uautomizer"
coveriteam/bin/coveriteam --cache-dir ./coveriteam/cache --no-cache-update cvt-files/programs/verifier.cvt --input verifier_path=cvt-files/actors/symbiotic-base.yml --input data_model=ILP32 --input specification_path=sv-benchmarks/c/properties/unreach-call.prp --input program_path=example.c

Here is an example output on the console after running the above command. The message says that Symbiotic was not able to verify the program within the CPU time limit.

Verifier :: symbiotic-svcomp23 :: {'program': 'Program', 'spec': 'Specification'} --> {'verdict': 'Verdict'}
2023-XX-XX XX:XX:XX WARNING The actor symbiotic-svcomp23 was terminated by BenchExec. Termination reason: cputime-soft. Possibly it did not produce the expected result.
{'verdict': 'KILLED (signal 15, verification)'}

Run a parallel portfolio of CPA-DF with another verifier

The following command executes a portfolio of CPA-DF and Symbiotic (Symbiotic || DF) via CoVeriteam and uses example.c as the input program.

# to use another verifier in the portfolio, replace "symbiotic" with "esbmc" or "uautomizer"
coveriteam/bin/coveriteam --cache-dir ./coveriteam/cache --no-cache-update cvt-files/programs/portfolio.cvt --input verifier_path=cvt-files/actors/symbiotic-par2.yml --input data_model=ILP32 --input specification_path=sv-benchmarks/c/properties/unreach-call.prp --input program_path=example.c

Here is an example output on the console after running the above command. The message shows that the portfolio was able to prove the program correct ({'verdict': 'true'}), and that Symbiotic was terminated because CPA-DF had already found a proof.

PORTFOLIO :: {'program': 'Program', 'spec': 'Specification'} --> {'verdict': 'Verdict'}
    Verifier :: cpachecker-local :: {'program': 'Program', 'spec': 'Specification'} --> {'verdict': 'Verdict'}
    Verifier :: symbiotic-svcomp23 :: {'program': 'Program', 'spec': 'Specification'} --> {'verdict': 'Verdict'}
2023-XX-XX XX:XX:XX WARNING Killing process 115457 forcefully.
2023-XX-XX XX:XX:XX WARNING The actor symbiotic-svcomp23 was terminated by BenchExec. Termination reason: killed. Possibly it did not produce the expected result.
{'verdict': 'true'}

View the experimental results

In paper-results/, you find the full experimental data presented in 3 HTML tables, where paper-results/cpachecker/results-df.table.html contains the results for RQ1 “Dynamic Precision Adjustment in CPA-DF”, paper-results/cpachecker/results-ki.table.html contains the results for RQ2 “Parallel Portfolio versus Invariant Injection”, and paper-results/coveriteam/results-boosting.table.html contains the results for RQ3 “Boosting Program Verification with CPA-DF”.

Here we provide the links to view the corresponding tables and figures of the paper. Please open the following with a browser:

• Table I: CPAchecker and CoVeriTeam
• Figure 1
• Figure 2
• Figure 3
• Figure 4

Reproduce the full experiments

Install BenchExec

We use BenchExec, a framework for reliable benchmarking and resource measurements, at version 3.16, to perform our evaluation. Please follow the installation guide to set up the benchmarking environment.

Experimental settings

The settings are described in the XML files bench-defs/*.xml. These XML files will be used by BenchExec for limiting computing resources and scheduling runs. For the execution of a task, a resource limit of 4 CPU cores, 900 seconds of CPU time, and 15 GB of memory is imposed.

Benchmark CPA-DF and other analyses in CPAchecker

To run the full experiment for RQ1 and RQ2, execute the following command.

benchexec bench-defs/cpachecker.xml --tool-directory cpachecker/ --read-only-dir / --overlay-dir . --hidden-dir /home/

Benchmark verifiers and portfolios in CoVeriTeam

To run the full experiment for RQ3, execute the following command.

benchexec bench-defs/coveriteam.xml --tool-directory coveriteam/ --read-only-dir / --overlay-dir . --hidden-dir /home/ --full-access-dir /sys/fs/cgroup/

Generate tables

After the execution of an experiment, a help message is shown on the console telling you how to generate a table for browsing the results.

[...redacted...]

In order to get HTML and CSV tables, run
table-generator results/coveriteam.2023-XX-XX_XX-XX-XX.results.esbmc.xml.bz2 results/coveriteam.2023-XX-XX_XX-XX-XX.results.esbmc-df.xml.bz2 results/coveriteam.2023-XX-XX_XX-XX-XX.results.symbiotic.xml.bz2 results/coveriteam.2023-XX-XX_XX-XX-XX.results.symbiotic-df.xml.bz2 results/coveriteam.2023-XX-XX_XX-XX-XX.results.uautomizer.xml.bz2 results/coveriteam.2023-XX-XX_XX-XX-XX.results.uautomizer-df.xml.bz2

Execute a subset of experiments with lower resource limits

The full experiments would require roughly 2 years of CPU time on a 3.40 GHz core. If you only want to execute a subset of verification tasks and with lower resource limits, add the following flags to the benchexec command:

# to execute another subset of tasks, replace "ReachSafety-BitVectors" with other tasks defined in bench-defs/*.xml
--timelimit 10s --memorylimit 1000MB --tasks ReachSafety-BitVectors

For more information about these flags, please refer to benchexec --help.

Known issues

Known issues of the artifact are documented in the README inside the archive.