Reproduction Package for TACAS 2024 Article `Btor2-Cert: A Certifying Hardware-Verification Framework Using Software Analyzers'
Description
Reproduction Package
Btor2-Cert: A Certifying Hardware-Verification Framework Using Software Analyzers
Abstract
This artifact is a reproduction package for the article “Btor2-Cert: A Certifying Hardware-Verification Framework Using Software Analyzers”, published in the proceedings of TACAS 2024. It is accessible via the DOI 10.5281/zenodo.10548597 on Zenodo.
The artifact consists of source code, precompiled tool executables, and input data used in the evaluation of the article, as well as the raw and processed results obtained from the experiments. Specifically, it contains (1) the source code of the proposed witness translator and the validator Btor2-Val, (2) the executables of the Btor2-to-C task translator Btor2C and the compared software verifiers and validators, (3) the benchmark suite of the word-level hardware-verification Btor2 tasks, and (4) the raw and processed data, including the verification witnesses and the translated files, generated from the evaluation.
The artifact aims to facilitate reproducing the evaluation reported in the publication. In particular, it supports the following claims: (1) The witness translator can translate software witnesses to Btor2 witnesses. (2) The validator Btor2-Val is effective and efficient compared to other mature validators. (3) The certifying verifier Btor2-Cert can deliver certified results on some Btor2 tasks that the state-of-the-art hardware model checkers fail to solve. For more information, please refer to the article.
To test the artifact conveniently, we selected a subset of the benchmark tasks that can be analyzed in roughly 2 hours if 1 CPU core and 4 GB of memory are allocated to the TACAS VM. Instructions for a kick-the-tires review pass can be found in the section TL;DR. To fully reproduce the experiments reported in the article, 2 CPU cores, 16 GB of memory, and roughly 50 days of CPU time are required.
Contents
This artifact contains the following items:
README.html
: this documentationLICENSE
: license information of the artifactverifiers/
: a directory containing the evaluated software verifiers, which are used for generating verification witnesses, and the hardware verifier ABC, which is used as a reference for comparisonvalidators/
: a directory containing the evaluated validators, which are used for validating the witnesses produced by software verifiersbtor2-val.zip
: an archive containing the proposed witness translator and the validator Btor2-Val
btor2-cert/
: the proposed certifying hardware verifier using software analyzersexamples/
: a directory containing the example verification tasks presented in the articlebenchmarks/
: a directory containing the word-level hardware-verification tasks used in our evaluationbench-defs/
: a directory containing the benchmark definitions for the experiments (used by BenchExec, a framework for reliable benchmarking)data-submission/
: a directory containing the raw and processed data produced from our full evaluation (used in the article, underpaper-results/
) and from a demo experiment (prepared for this reproduction package, underdemo-results/
)packages/
: a directory containing the necessary Debian and Python packages to set up the environment for the experiments in the TACAS ’23 Artifact Evaluation VMscripts/
: a directory containing utility scriptsMakefile
: a recipe for assembling commands to run experiments and process data
This README file will guide you through the following steps:
- Set up evaluation environment
- Execute Btor2-Cert
- Perform evaluation
- Analyze experimental data
- Known issues of the artifact
TL;DR
On the TACAS ’23 Artifact Evaluation (AE) VM, type the following commands in the root folder of the reproduction package to:
-
Install required dependencies (root permission required):
make install-deb-packages
: install required Debian packagesmake install-py-packages
: install required Python packages
-
Set up BenchExec (executable inside
validators/btor2-val.zip
; root permission required):make configure-cgroups configure-benchexec-permission
: configure cgroups and other permissions for BenchExec- WARNING: the script changes the version of cgroups to version 1
- Important: please reboot your system afterwards for the settings to take effect!
make prepare-benchexec
: turn off swap memory and allow user namespaces (needs to be redone after every reboot)make check-cgroups
: check if cgroups are configured correctly
-
Test Btor2-Cert with its default configurations (using CPAchecker’s predicate abstraction; Btor2-Val accepting invariants) on the safe and unsafe versions of the Btor2 circuit in Fig. 2 of the article:
make input-btor2=examples/example-safe.btor2 run-btor2cert
: execute Btor2-Cert (expected result:The verification result 'true' is certified and validated
, meaning that a proof was found and validated)make input-btor2=examples/example-unsafe.btor2 run-btor2cert
: execute Btor2-Cert (expected result:The verification result 'false' is certified and validated
, meaning that a bug was found and validated)
-
Test the software verifiers used by Btor2-Cert:
make input-c=examples/example-unsafe.c run-cbmc-bmc
: execute CBMC (expected result:FALSE
, meaning that a bug was found)make input-c=examples/example-unsafe.c run-esbmc-bmc
: execute ESBMC (expected result:FALSE_REACH
, meaning that a bug was found)make input-c=examples/example-safe.c run-cpachecker-predabs
: execute CPAchecker (expected result:Verification result: TRUE
, meaning that the program is safe)make input-c=examples/example-safe.c run-uautomizer-default
: execute Ultimate Automizer (expected results:Result: TRUE
, meaning that the program is safe)
-
Test SW-to-HW witness translation and validator Btor2-Val used by Btor2-Cert:
make input-btor2=examples/example-safe.btor2 input-c=examples/example-safe.c input-witness=examples/example-safe.witness.graphml run-btor2val-correctness
: execute Btor2-Val for correctness witness validation (expected result:Validation result: true(valid invariant)
, meaning that the correctness witness is confirmed)make input-btor2=examples/twocount32.btor2 input-c=examples/twocount32.c input-witness=examples/twocount32.witness.graphml run-btor2val-violation
: execute Btor2-Val for violation witness validation (expected result:Validation result: false
, meaning that the violation witness is confirmed)
-
Test other evaluated software witness validators:
make input-c=examples/example-safe.c input-witness=examples/example-safe.witness.graphml run-liv
: execute LIV (expected result:Overall result: true
, meaning that the correctness witness is confirmed)make input-c=examples/twocount32.c input-witness=examples/twocount32.witness.graphml run-cpaw2t
: execute CPA-w2t (expected result:Verification result: FALSE
, meaning that the violation witness is confirmed)make input-c=examples/twocount32.c input-witness=examples/twocount32.witness.graphml run-fshellw2t
: execute FShell-w2t (expected result:FALSE
, meaning that the violation witness is confirmed)
-
Perform demo experiments:
- Make sure that
make check-cgroups
displaysCheck passed
; if it shows errors or warnings, usemake configure-cgroups
to fix them make demo-verify-and-validate-safe
: invoke Btor2-Cert with CPAchecker for verification and Btor2-Val and LIV for validation on 30 selected safe tasksmake demo-verify-and-validate-unsafe
: invoke Btor2-Cert with ESBMC for verification and Btor2-Val and FShell-w2t for validation on 30 selected unsafe tasks
To quickly check the experiment is runnable, you could use
make timelimit=10s memlimit=1GB cpulimit=1 demo-verify-and-validate-unsafe
to set the run-time limit to 60 seconds, memory limit to 1 GB, and CPU limit to 1 core per task. After the runs are finished, HTML tables containing the experimental results can be found inexp-results/tables/{correctness,violation}/demo.*.validated.table.html
. - Make sure that
-
Perform full evaluation (needs 2 CPU cores, 16 GB of RAM, and roughly 50 days of CPU time):
make verify-and-validate-safe
: on 758 safe tasksmake verify-and-validate-unsafe
: on 456 unsafe tasks
After the run is finished, HTML tables containing the experimental results can be found in
exp-results/tables/{correctness,violation}/tab{2,3}.*.table.html
. -
To view HTML files corresponding to tables and figures of the paper, please open the following links with a browser:
- Table 2 (the lower half)
- Table 3 (the lower half)
- Figure 8(a)
- Figure 8(b)
In the following we only excerpt important fragments of the log files to improve readability. The complete log messages for the mentioned commands are listed in data-submission/complete-logs.html for your reference.
Set Up Evaluation Environment
Hardware Requirements
Please provide at least 4 GB of memory and 1 CPU core for the demo experiments. The full experiments need 16 GB of memory and 2 CPU cores. An internet connection is not necessary.
Software Requirements
This artifact requires a Linux-based operating system using cgroups v1 and has been tested on a 64-bit Ubuntu 22.04 computer with Linux kernel 5.15.0.
To reproduce the evaluation in the article, we need additional dependencies listed below:
On the TACAS ’23 AE VM, these requirements can be installed via the following command (you will be asked for root permission):
make install-deb-packages install-py-packages
If you are not using the TACAS ’23 AE VM, please install these dependencies from your distribution.
Set Up BenchExec
We use BenchExec, a framework for reliable benchmarking and resource measurements, to perform our evaluation.
To set up BenchExec (executable in validators/btor2-val.zip
), please run:
make configure-cgroups configure-benchexec-permission
WARNING: The script will change the version of cgroups to version 1. If you do not want this change on your machine, we recommend testing the reproduction package in the TACAS ’23 AE VM.
Important: After running the above command, please reboot your system for the settings to take effect!
Note that an additional preparation for BenchExec is required after each reboot. Please run (you will again be asked for root permission):
make prepare-benchexec
The above command turns off the swap memory and allows user namespaces to be used.
To test if the permission of cgroups needed by BenchExec is correctly configured, please run:
make check-cgroups
No warnings or error messages should be printed if the permission is correctly configured.
If there are still unresolved problems, please take a look at BenchExec’s installation guide.
Execute Btor2-Cert
The proposed certifying hardware verifier Btor2-Cert performs the following steps on a Btor2 verification task:
- translating the verification tasks from a Btor2 circuit to a C program using Btor2C,
- verifying the C program and generating a software witness using a software verifier,
- translating the software witness to a Btor2 witness, and
- validating the Btor2 witness using Btor2-Val.
To produce a certified and validated verification result on a Btor2 circuit using Btor2-Cert, please run:
make input-btor2=<Btor2-file> run-btor2cert
The above command will invoke Btor2-Cert’s default configurations, using CPAchecker’s predicate abstraction as the verification backend and setting Btor2-Val to check for valid invariants (cf. invariant quality in the article).
The used backend software verifier and its version/configuration as well as the invariant quality checked by Btor2-Val can be specified via the argument btor2cert-args
. Currently, the following verification configurations are supported:
cpachecker:2.2.1-svn-44619-predabs
(default)cpachecker:2.2.1-svn-44619-impact
cpachecker:2.2.1-svn-44619-imc
cpachecker:2.2.1-svn-44619-ismc
cpachecker:2.2.1-svn-44619-bmc
cbmc:svcomp23
esbmc:svcomp23-bmc
uautomizer:0.2.3-6fd36663
The three levels of invariant quality for a candidate invariant Inv are as follows:
valid
: Inv is an invariant (default)safe
: Inv is a safe invariantinductive
: Inv is a safe and inductive invariant
For example, to verify the program example/example-safe.btor2
with the Impact algorithm in CPAchecker and configure Btor2-Val to accept safe invariants, run
make input-btor2=examples/example-safe.btor2 \
btor2cert-args="--verifier cpachecker:2.2.1-svn-44619-impact --level safe" \
run-btor2cert
The certified and validated verification result is printed on the console:
[INFO] Checking validity (M |= Inv): ok
[INFO] Checking safety (Inv => P): ok
[INFO] Validation result: true(safe invariant)
[INFO] The verification result 'true' is certified and validated
The tool outputs, including the translated C program, verification witness, and log files, are written to folder output/
.
For more information, please refer to the help message by running ./btor2-cert/btor2-cert -h
or its documentation.
In the following, we explain the details of the used software verifiers, Btor2-Val, and the compared validators.
Certifying Software Verifiers
All verifiers used in the evaluation have been precompiled and archived. The corresponding tool archives can be found in verifiers/*.zip
. The sources of each tool are listed in verifiers/source.md
.
The following make targets (run-<verifier>-<config>
) can be used to invoke a specific configuration of a verifier:
run-cbmc-bmc
run-cpachecker-bmc
run-cpachecker-imc
run-cpachecker-ismc
run-cpachecker-impact
run-cpachecker-predabs
run-esbmc-bmc
run-uautomizer-default
For example, to verify the program example-safe.c
with predicate abstraction of CPAchecker, please run:
make input-c=examples/example-safe.c run-cpachecker-predabs
The input program to the verifier is specified via the argument input-c=path/to/c-program
. Another program example-unsafe.c
can also be used for testing the verifiers.
The (case-insensitive) verification verdict is printed on the console during execution, where
TRUE
means the program satisfies the safety property,FALSE
(orFALSE_REACH
) means a property violation is found, andUNKNOWN
means that the verifier cannot determine the result (due to unsupported features or insufficient computing resource, etc.).
Note that the configurations running bounded model checking (*-bmc
) can only identify violations but not prove the safety property.
Along with a verdict TRUE
(resp. FALSE
), a verifier generates a correctness (resp. violation) witness, which serves as a certificate of the verification result. The location of the witness file is tool-dependent but can be tracked down by
# replace "cpachecker" with "cbmc", "esbmc", or "uautomizer"
find verifiers/cpachecker/ -type f -name *.graphml
Witness Validators
The evaluated validators have been precompiled and archived. The corresponding tool archives can be found in validators/*.zip
. The sources of each tool are listed in validators/source.md
.
Depending on the type of a witness (correctness or violation), the following make targets can be used for witness validation and invoke different validators.
- Validating correctness witnesses:
run-btor2val-correctness
run-liv
run-uautomizer-val
- Validating violation witnesses:
run-btor2val-violation
run-cpaw2t
run-fshellw2t
Witness Translation and Btor2-Val
As input, Btor2-Val takes a Btor2 circuit, the corresponding C program translated by Btor2C, and the software witness (a protocol automaton in the GraphML format) produced by a software verifier. It first translates a software witness to a Btor2 witness then validates it with a hardware verifier or simulator.
To validate the correctness witness example-safe.witness.graphml
of the program example-safe.c
, run
make input-btor2=examples/example-safe.btor2 \
input-c=examples/example-safe.c \
input-witness=examples/example-safe.witness.graphml \
run-btor2val-correctness
Similarly, to validate the example violation witness, replace the arguments given to input-{btor2,c,witness}
with example-unsafe.{btor2,c,witness.graphml}
and the make target with run-btor2val-violation
, respectively.
Additional arguments can be relayed to Btor2-val via btor2val-args
. For instance, to print the help messages of Btor2-Val, run
make btor2val-args=-h run-btor2val-correctness
The levels of invariant quality for a candidate invariant Inv can be configured by specifying the following arguments in btor2val-args
.
"--level valid"
: Inv is an invariant (default)"--level safe"
: Inv is a safe invariant"--level inductive"
: Inv is a safe and inductive invariant
For more information, please refer to the documentation of Btor2-Val.
Other Software Witness Validators
The other evaluated validators take a C program and a software witness as input.
For example, to validate the correctness witness example-safe.witness.graphml
of the program example-safe.c
using LIV, run
make input-c=examples/example-safe.c \
input-witness=examples/example-safe.witness.graphml \
run-liv
Neither CPA-w2t nor FShell-w2t is able to validate the violation witness example-unsafe.witness.graphml
(produced by CPAchecker) for the program example-unsafe.c
due to some technical issues of the validators. Therefore, we provide another pair of program twocount32.c
and witness twocount32.witness.graphml
for testing them. To invoke CPA-w2t or FShell-w2t, run
make input-c=examples/twocount32.c \
input-witness=examples/twocount32.witness.graphml \
run-cpaw2t # or run-fshellw2t
Understand Validation Results
The (case-insensitive) validation verdict is printed on the console at the end of an execution.
For a correctness witness,
TRUE
indicates that the witness is confirmed, andFALSE
orUNKNOWN
indicate that the witness cannot be validated, either because the candidate invariant does not meet the required invariant quality, the computing resource is insufficient, or some other errors.
For a violation witness,
FALSE
indicates that the witness is confirmed, andTRUE
orUNKNOWN
indicate that the witness cannot be validated, either due to the incompleteness of the error path in the witness, insufficient computing resource, or some other errors.
In short, a verification witness is confirmed or the verification result is certified if the verdicts obtained by the verifier and the validator agree.
Perform Evaluation
The experimental results, including raw and processed data, are stored under exp-results/
. Before starting a new set of experiments, please make sure to remove the folder exp-results/
.
A full reproduction of the article’s evaluation takes roughly 50 days of CPU time. The required time for each step is listed below (time unit: day).
safe tasks | unsafe tasks | |
---|---|---|
verification | 35.5 | 11.4 |
validation | 1.9 | 1.4 |
We provide two settings for the experiments: one for demonstration and the other for the full evaluation. The two settings differ in (1) the set of executed tasks and (2) the executed tools/algorithms. All the other common settings are explained below.
Experimental Settings
The settings are described in the XML files bench-defs/{verification,validation}/{correctness,violation}/*.xml
. These XML files are used by BenchExec, a framework for reliable benchmarking.
For the execution of a task, a default resource limit of 2 CPU cores, 900 seconds of CPU time, and 15 GB of memory is imposed. (If the required resource is not available on your system, please follow the instructions explained below to adjust the limit.)
The XML files contain the following configurations of the compared verifiers and validators in the evaluation, namely:
- Verifiers
- Validators
Before you start executing any experiment, please make sure that
- BenchExec is set up successfully (instructions) and
- cgroups is correctly configured by running
make check-cgroups
.
If make check-cgroups
reports errors or warnings, type make prepare-benchexec
to fix them.
Demo Experiments
A complete experiment on the whole benchmark suite, consisting of 758 and 456 safe and unsafe tasks (listed in benchmarks/bv/btor2/bv64-{true,false}-tasks.set
respectively), takes a vast amount of time. The experimental data produced from the full evaluation reported in the paper can be found in folder data-submission/paper-results/
.
To show how our experiments were conducted, we selected 30 safe and 30 unsafe tasks from the benchmark suite (listed in bench-defs/sets/demo-{true,false}.set
), 2 verifiers (CPAchecker and ESBMC), and 3 validators (Btor2-Val, LIV, and FShell-w2t) for demonstration.
We emphasize that this experiment is only for demonstration purposes. The observations on the comparison between tools in the article were drawn from the evaluation on the whole benchmark suite.
This demonstrative experiment was designed such that it is feasible given reasonable hardware equipment and time: It could be finished within 1.5 hours on a regular laptop.
To start the demo experiments, run the command below:
make demo-verify-and-validate-safe demo-verify-and-validate-unsafe
Below is an example on how to adjust the resource limits. Suppose you would like to set the time limit to 60 seconds, the memory limit to 3 GB, and use only 1 CPU core for a task, please run:
make timelimit=60s memlimit=3GB cpulimit=1 \
demo-verify-and-validate-safe demo-verify-and-validate-unsafe
Moreover, if you have enough hardware resources and would like to launch parallel benchmark tasks, add num-jobs=INT
to the make command. Note that running experiments with lower resource limits could lead to timeout or OOM for some tasks.
After the experiments are finished, HTML tables containing the experimental results can be found in exp-results/tables/
.
Full Experiments
To perform all the experiments reported in the article, run:
make verify-and-validate-safe # verification and validation on safe tasks
make verify-and-validate-unsafe # verification and validation on unsafe tasks
make compare-with-abc # benchmark the hardware verifier ABC
Note that the full evaluation will produce around 20 GB data for verification witnesses.
Please make sure that you have enough disk space.
The three make targets are composed by multiple nested sub-targets as shown below.
verify-and-validate-safe
verify-safe
verify-safe-cpachecker
verify-safe-uautomizer
validate-safe
validate-safe-btor2val-valid
validate-safe-btor2val-safe
validate-safe-btor2val-inductive
validate-safe-liv
validate-safe-uautomizer
verify-and-validate-unsafe
verify-unsafe
verify-unsafe-cbmc
verify-unsafe-cpachecker
verify-unsafe-esbmc
verify-unsafe-uautomizer
validate-unsafe
validate-unsafe-btor2val
validate-unsafe-btor2val-fromlog
validate-unsafe-cpaw2t
validate-unsafe-fshellw2t
compare-with-abc
verify-safe-abc
verify-unsafe-abc
Each target can be executed separately. Therefore, you can select the corresponding make target to benchmark the tool in which you are interested. However, please note that validation can only be performed after verification, i.e., after witnesses are generated. As verification runs are typically more time-consuming, a helpful target link-verification-results
is provided to unzip and link the verification witnesses produced from our experiments to the working directory of current experiments. (The unzipped witness files take up around 20 GB in total.) That is, one can conduct validation-only experiments by running:
make link-verification-results validate-safe validate-unsafe
After the experiments are finished, HTML tables containing the experimental results can be found in exp-results/tables/
.
Analyze Experimental Data
We recommend taking advantage of the interactive HTML files to help visualize the results of the experiments. These files can be easily opened with a web browser (e.g., firefox), and can display the information presented in all tables and figures of the article.
Results from Our Experiments
The results (both raw and processed data) of full and demo experiments obtained by our machines are in folders data-submission/{paper,demo}-results/
. The full experiments were performed to collect the data used in the paper, and the demo experiments were performed in order to prepare this artifact
The generated HTML tables (under data-submission/{paper,demo}-results/tables/{correctness,violation}/
) are:
-
Full experiments
HTML table Tasks Verifiers Validators Fig./Table RQs Make target tab2.cpachecker-imc.validated.table.html
safe CPAchecker-IMC Btor2-Val, LIV Table 2 1~3 verify-and-validate-safe
tab2.cpachecker-ismc.validated.table.html
safe CPAchecker-ISMC Btor2-Val, LIV Table 2 1~3 verify-and-validate-safe
tab2.cpachecker-impact.validated.table.html
safe CPAchecker-Impact Btor2-Val, LIV Table 2 1~3 verify-and-validate-safe
tab2.cpachecker-predabs.validated.table.html
safe CPAchecker-PredAbs Btor2-Val, LIV Table 2, Fig. 8(a) 1~4 verify-and-validate-safe
tab2.uautomizer-default.validated.table.html
safe UAutomizer Btor2-Val, LIV, UAutomizer Table 2 1~3 verify-and-validate-safe
abc-pdr-comparison.table.html
safe ABC-PDR, CPAchecker-PredAbs Btor2-Val - 5 compare-with-abc
tab3.cbmc-bmc.validated.table.html
unsafe CBMC Btor2-Val, {CPA,FShell}-w2t Table 3 1~3 verify-and-validate-unsafe
tab3.cpachecker-bmc.validated.table.html
unsafe CPAchecker-BMC Btor2-Val, {CPA,FShell}-w2t Table 3, Figure 8(b) 1~4 verify-and-validate-unsafe
tab3.esbmc-bmc.validated.table.html
unsafe ESBMC Btor2-Val, {CPA,FShell}-w2t Table 3 1~3 verify-and-validate-unsafe
tab3.uautomizer-default.validated.table.html
unsafe UAutomizer Btor2-Val, {CPA,FShell}-w2t Table 3 1~3 verify-and-validate-unsafe
abc-bmc-comparison.table.html
unsafe ABC-BMC, CBMC, CPAchecker-BMC, ESBMC Btor2-Val - 5 compare-with-abc
-
Demo experiments
HTML table Tasks Verifier Validators Make target demo.cpachecker-predabs.validated.table.html
safe CPAchecker-PredAbs Btor2-Val, LIV demo-verify-and-validate-safe
demo.esbmc-bmc.validated.table.html
unsafe ESBMC Btor2-Val, FShell-w2t demo-verify-and-validate-unsafe
We also provide pre-configured links to easily view the exact tables/figures as shown in the paper, as listed in the TL;DR section.
Navigate Through the Data
Once a collection of experiments ({demo-,}verify-and-validate-{safe,unsafe}
and compare-with-abc
) is finished, the Makefile automatically aggregates the verification and validation results and generates the HTML files under exp-results/tables/
. If you would like to generate HTML tables alone for some other experiments, e.g., for verify-unsafe-cbmc
, please follow the instructions printed on the console (an example is shown below). Note that the path to table-generator
is <artifact-root>/validators/btor2-val/lib/benchexec/bin/table-generator
, please adjust it accordingly.
In order to get HTML and CSV tables, run
../../validators/btor2-val/lib/benchexec/bin/table-generator
/home/tacas23/artifact/exp-results/verification/violation/cbmc.2024-XX-XX_XX-XX-XX.results.cbmc-bmc.bv64-unsafe.xml.bz2
When opening the generated HTML table, you will be guided to the Summary
page of the experiment, where detailed settings of the experiment and a summary table of the compared tools/algorithms are displayed. For example, if you open tab3.cpachecker-bmc.validated.table.html
, in this page you can see the number of tasks listed in column “CPAchecker” of Table 3.
To inspect the log file of an individual task, navigate to the tab Table
and click on the status of that task. If the log file cannot be displayed, configure your browser according to the printed instructions.
To filter tasks, you can make use of the task filter at the upper-right corner of the page. To view quantile plots, please navigate to tab Quantile Plot
and adjust the drop-down menus as you prefer. To view scatter plots, please navigate to tab Scatter Plot
, and adjust the x- and y- axes according to your interests.
Known Issues of the Artifact
Known issues of this artifact are documented below.
CPU-throttling Warnings
When running experiments (especially on a laptop), BenchExec might raise the following warning:
2024-XX-XX XX:XX:XX - WARNING - CPU throttled itself during benchmarking due to overheating. Benchmark results are unreliable!
This is normal on a laptop. Please ignore it.
File-not-found warnings
When running validation experiments (make targets validate-*
), BenchExec might raise the following warning:
2024-XX-XX XX:XX:XX - WARNING - No files found matching
‘../../../exp-results/verification/violation/uautomizer.2024-XX-XX_XX-XX-XX.files/${rundefinition_name}/${taskdef_name}/witness.graphml’.
2024-XX-XX XX:XX:XX - WARNING - Pattern
../../../exp-results/verification/violation/uautomizer.2024-XX-XX_XX-XX-XX.files/${rundefinition_name}/${taskdef_name}/witness.graphml
in requiredfiles tag did not match any file for task
../../benchmarks/bv/c-eagerMod/goel-opensource/h_b04.yml.
The reason is that the shown verifier (in this example UAutomizer) did not produce a witness for the shown task (in this example h_b04
) during the previous verification runs, and thus BenchExec could not locate the witness file. This could happen if the verification result was UNKNOWN
for the corresponding task or the task has not been given to the verifier (in demo experiments, only a subset of verifiers and tasks are executed). Therefore, these warnings can be safely ignored.
Files
Btor2Cert-artifact-TACAS24-proceedings.zip
Files
(3.2 GB)
Name | Size | Download all |
---|---|---|
md5:caa3af839483bc8781249b21d0aec15b
|
3.2 GB | Preview Download |
Additional details
Dates
- Available
-
2023-10-23Reproduction Package, TACAS24-submission
- Available
-
2024-04-03Reproduction Package, TACAS24-proceedings