Reproduction Package for Article `Construction of Verifier Combinations Based on Off-the-Shelf Verifiers'

This file describes the contents of the replication package for the article “Construction of Verifier Combinations Based on Off-the-Shelf Verifiers”, submitted to FASE 2022. It contains the experimental data for the evaluation done for the article.

Authors

Anonymous Authors

License:

SPDX-License-Identifier: CC-BY-4.0
https://spdx.org/licenses/CC-BY-4.0.html

Different directories of this artifact are licensed differently. For details see LICENSE.

Contents of this Archive:

    .
    |-- benchdefs/                       # benchmark definition files used to execute the experiments
    |-- benchexec/                       # submodule of the BenchExec project: https://github.com/sosy-lab/benchexec
    |-- coveriteam/                      # submodule of the CoVeriTeam project: https://gitlab.com/sosy-lab/software/coveriteam
        |-- cache/                       # directory containing all the downloaded tools that are used in the experiments 
    |-- coveriteam-input/                # input data to  CoVeriTeam programs used in evaluation
        |-- actor-definitions/           # actor definitions used in the experiments; some of it are generated
        |-- cvt-programs/                # CoVeriTeam programs used in the experiments
    |-- hardness_models/                 # hardness models
        |-- checkpoints/default/         # The default selection checkpoint used in evaluation
        |-- data/                        # Precomputed embeddings and labels for SVComp 2021
        |-- train_hardness_model.ipynb   # notebook used for description and replication of the training process
    |-- results/                         # results of the execution
        |-- raw/                         # result files produced by execution of the benchmarks set
        |-- html-tables/                 # html-tables produced by the `table-generator`
        |-- index.html                   # web page describing experiment results and containing links to result tables.
    |-- svcomp21-results/                # XML result files of the standalone tools; taken from the software verification competition 2021: https://sv-comp.sosy-lab.org/2021/
    |-- tabledefs/                       # table definition files used by the `table-generator` to create the html tables
    |-- Dockerfile.user.2021             # Dockerfile configuration taken from the software verification competition 2021: https://sv-comp.sosy-lab.org/2021/
    |-- LICENSE.txt                      # file containing license information
    |-- Makefile                         # Makefile containing targets to create the additional input files, benchmark templates, table definition file and html tables.

System requirements

The experiments could be reproduced on the machine running Ubuntu 20.04 on which the packages listed in Dockerfile.user.2021 have been installed.

Details about the directories in this archive

• benchdefs/: This directory contains the benchmark definition files that are input to the tool BenchExec. More details about the benchmark definition files are available at: https://github.com/sosy-lab/benchexec/blob/main/doc/benchexec.md
• benchexec/: This directory contains the tool BenchExec:https://github.com/sosy-lab/benchexec. We have taken the commit: 344eb314bad07142a8ff0cebc2eae9502470ef60
• coveriteam/: This directory contains the tool CoVeriTeam: https://gitlab.com/sosy-lab/software/coveriteam. We have taken the commit: a48227a67d67950c022f9f41c1b2d0e7778343b9. The tool combinations evaluated in the article have been constructed using CoVeriTeam.
  • cache/: CoVeriTeam downloads the tool archives in this directory. This directory contains all the archives of the tools that have been used in the evaluation.
• coveriteam-input/: This directory contains the input data for the tool CoVeriTeam for constructing the tool combinations.
  • actor-definitions/: Actor definitions in YAML format. An actor definition is used by CoVeriTeam to create actors. More information is available at: coveriteam/doc/index.md#actor-definition
  • cvt-programs/: Program describing the tool combinations. These are input to CoVeriTeam to create tool combinations.
• hardness_models/: data related to hardness models.
  • checkpoints/default/: The selection checkpoints used in the experimental evaluation in the article.
  • data/: Precomputed embeddings and labels for SVComp 2021 used to train the hardness models.
  • train_hardness_model.ipynb: Jupyter notebook used for description and replication of the training process of the hardness models.
• results/: directory containing the results of the experiments conducted for evaluation.
  • raw/: contains the raw result files of the experiments, as produced by BenchExec. More information is available at: https://github.com/sosy-lab/benchexec/blob/main/doc/run-results.md.
  • html-tables/: contains the html-tables produced by executing the table-generator on the result files. More information about the table-generator is available at: https://github.com/sosy-lab/benchexec/blob/main/doc/table-generator.md.
  • index.html: this is an html file describing the experiments and their results. It can be opened simply in a browser, but in this case the viewer won’t be able to access the logs. It is better to simply start a php server by executing php -S 127.0.0.1:8000 and browse http://127.0.0.1:8000. This way one can view the logs in the html tables.
• svcomp21-results/: results of the standalone tools from the software competition 2021. We compare the results of the compositions against these results.
• tabledefs/: table definition templates used to generate the html-tables. These files are consumed by table-generator to generate the tables.
• Dockerfile.user.2021: file describing the packages that need to be installed on an Ubuntu 20.04 machine to execute the tools. This file is provided by the organizers of the software verification competition.
• LICENSE.txt: file containing the license information
• Makefile: Makefile used to partially generate the data for, and execute the benchmarking tasks for the evaluation.

Benchmark tasks

The experiments were executed on the Benchmark Set of 10th Intl. Competition on Software Verification (SV-COMP 2021): https://zenodo.org/record/4459126