HisFuzzer

Artifacts for the ISSTA submission #502

Table of Contents

• Background
• Install
• Usage
• Reports

 

Background

HisFuzzer utilizes historical data to test SMT solvers.

You can use it to collect the bug-triggering formulas from Z3 and cvc5's bug tracking systems.

You can also perform differential tests on the SMT solvers.

See below for more details on the usage.

 

Install

This project uses efficient-apriori, antlr4, z3 API, and github3. Go check them out if you don't have them locally installed.

pip3 install efficient-apriori 

pip3 install antlr4-python3-runtime==4.9.2

pip3 install z3-solver

pip3 install github3.py

Besides, if you want to test the solvers, you need to download and build them first.

You can download Z3 and cvc5 in Z3Prover/z3: The Z3 Theorem Prover (github.com) and cvc5/cvc5: cvc5 is an open-source automatic theorem prover for Satisfiability Modulo Theories (SMT) problems. (github.com).

Please follow the instructions given by them for installation.

 

Usage

You can use HisFuzzer to update the historical data in the bug tracking systems of Z3 and cvc5 with your GitHub personal access token:

bin/HisFuzzer --update --token=[TOKEN]

GitHub personal access token can be generated in Personal Access Tokens (github.com).

In general, the first update process takes a long time.

 

Since we have already initialized it, you can use it directly without updating to test the solvers, for example:

bin/HisFuzzer --solver1=z3 --solverbin1=/home/z3/build/z3 --solver2=cvc5 --solverbin2=/home/cvc5/build/bin/cvc5 --benchmark=/home/benchmarks

You can download the benchmarks in SMT-LIB-benchmarks · GitLab (uiowa.edu), and you can use the above command to mutate the .smt2 files in the downloaded benchmarks to try to find the bugs.

 

In addition, we also give an example that can be easily reproduced for you to experience:

bin/HisFuzzer --solver1=z3 --solverbin1=/home/z3/build/z3 --solver2=cvc5 --solverbin2=/home/cvc5/build/bin/cvc5 --reproduce=bug

Note that when reproducing, use git checkout to switch to the appropriate version before building the SMT solver.

Please refer to the readme.md file in the /reproduce folder for more details.

We will add more easily reproducible bugs for you to experience later. Please stay tuned for the latest version.

 

Bug reports (updating)

z3

https://github.com/Z3Prover/z3/issues/5722#issuecomment-998452982 (fixed) 

https://github.com/Z3Prover/z3/issues/5737 (moved to https://github.com/Z3Prover/z3/issues/4889#issuecomment-1004329224) (confirmed)

https://github.com/Z3Prover/z3/issues/5753#issuecomment-1007087883 (duplicate)

https://github.com/Z3Prover/z3/issues/5753#issuecomment-1008214504 (invalid)

https://github.com/Z3Prover/z3/issues/5774 (invalid)

https://github.com/Z3Prover/z3/issues/5777#issuecomment-1014065487 (fixed)

https://github.com/Z3Prover/z3/issues/5722#issue-1084089459 (fixed) 

https://github.com/Z3Prover/z3/issues/5798 (reported)

https://github.com/Z3Prover/z3/issues/5769 (confirmed)

https://github.com/Z3Prover/z3/issues/5769#issuecomment-1015129354 (reported)

https://github.com/Z3Prover/z3/issues/5778#issuecomment-1018132484 (fixed)

https://github.com/Z3Prover/z3/issues/5789 (fixed) 

https://github.com/Z3Prover/z3/issues/5806 (reported)

https://github.com/Z3Prover/z3/issues/5722#issuecomment-998652655 (invalid)

https://github.com/Z3Prover/z3/issues/5732 (fixed) 

https://github.com/Z3Prover/z3/issues/5734 (moved to https://github.com/Z3Prover/z3/issues/5641#issuecomment-1000497502) (fixed)

https://github.com/Z3Prover/z3/issues/5641#issuecomment-1000804719 (moved to https://github.com/Z3Prover/z3/issues/4889#issuecomment-1004328993) (confirmed)

https://github.com/Z3Prover/z3/issues/5741 (fixed)

https://github.com/Z3Prover/z3/issues/5753#issuecomment-1007067681 (duplicate)

https://github.com/Z3Prover/z3/issues/5754 (duplicate)

https://github.com/Z3Prover/z3/issues/5758 (fixed) 

https://github.com/Z3Prover/z3/issues/5753#issuecomment-1007982562 (fixed) 

https://github.com/Z3Prover/z3/issues/5770 (invalid)

https://github.com/Z3Prover/z3/issues/5753#issuecomment-1011732992 (fixed)

https://github.com/Z3Prover/z3/issues/5753#issuecomment-1011749964 (fixed)

https://github.com/Z3Prover/z3/issues/5753#issuecomment-1011755733 (fixed) 

https://github.com/Z3Prover/z3/issues/5753#issuecomment-1012024921 (fixed) 

https://github.com/Z3Prover/z3/issues/5779 (invalid)

https://github.com/Z3Prover/z3/issues/5778#issuecomment-1016018402 (fixed)

https://github.com/Z3Prover/z3/issues/5784 (fixed)

https://github.com/Z3Prover/z3/issues/5804 (fixed)

 

cvc5

https://github.com/cvc5/cvc5/issues/7924 (fixed) 

https://github.com/cvc5/cvc5/issues/7938 (confirmed)

https://github.com/cvc5/cvc5/issues/8016 (fixed)

https://github.com/cvc5/cvc5/issues/7858 (duplicate)

https://github.com/cvc5/cvc5/issues/7858#issuecomment-1000794259 (duplicate)

https://github.com/cvc5/cvc5/issues/7893 (confirmed)

https://github.com/cvc5/cvc5/issues/7894 (fixed) 

https://github.com/cvc5/cvc5/issues/7902 (fixed)

https://github.com/cvc5/cvc5/issues/7917 (fixed)

https://github.com/cvc5/cvc5/issues/7918 (fixed) 

https://github.com/cvc5/cvc5/issues/7919 (duplicate)

https://github.com/cvc5/cvc5/issues/7937 (fixed) 

https://github.com/cvc5/cvc5/issues/7974 (fixed)

https://github.com/cvc5/cvc5/issues/8001 (fixed)

https://github.com/cvc5/cvc5/issues/8052 (fixed)

https://github.com/cvc5/cvc5/issues/8067 (fixed)

https://github.com/cvc5/cvc5/issues/8094 (fixed)

https://github.com/cvc5/cvc5/issues/8111 (fixed)

https://github.com/cvc5/cvc5/issues/8096#issuecomment-1043800815 (fixed)

https://github.com/cvc5/cvc5/issues/8119 (fixed)

https://github.com/cvc5/cvc5/issues/8119#issuecomment-1045646303 (fixed)

https://github.com/cvc5/cvc5/issues/8124 (fixed)

https://github.com/cvc5/cvc5/issues/8133 (confirmed)

https://github.com/cvc5/cvc5/issues/8118 (fixed)

https://github.com/cvc5/cvc5/issues/8147 (fixed)

https://github.com/cvc5/cvc5/issues/8148#issuecomment-1049758826 (fixed)

https://github.com/cvc5/cvc5/issues/8162 (fixed)

https://github.com/cvc5/cvc5/issues/8163 (fixed)

https://github.com/cvc5/cvc5/issues/8158 (reported)

https://github.com/cvc5/cvc5/issues/8160 (fixed)