HistFuzz-test

Artifacts for the ICSE submission #20

Table of Contents

• Background
• Install
• Usage
• Data
• Reports

 

Background

HistFuzz utilizes historical data to test SMT solvers.

You can use it to collect the bug-triggering formulas from Z3 and cvc5's bug tracking systems.

You can also use HistFuzz to perform differential testing on the SMT solvers.

See below for more details on the usage.

 

Install

This project uses efficient-apriori, antlr4, and github3. Go check them out if you don't have them locally installed.

pip3 install efficient-apriori 

pip3 install antlr4-python3-runtime==4.9.2

pip3 install github3.py

Besides, if you want to test the solvers, you need to download and build them first.

You can download Z3 and cvc5 in Z3Prover/z3: The Z3 Theorem Prover (github.com) and cvc5/cvc5: cvc5 is an open-source automatic theorem prover for Satisfiability Modulo Theories (SMT) problems. (github.com).

Please follow their instructions for installation.

 

Usage

You can use HistFuzz to update the historical data in the bug tracking systems of Z3 and cvc5 with your GitHub personal access token:

bin/HistFuzz --update --token=[TOKEN]

GitHub personal access token can be generated in Personal Access Tokens (github.com).

 

Since we have already initialized it, you can use it directly without updating to test the solvers, for example:

bin/HistFuzz --solver1=z3 --solverbin1=/home/z3/build/z3 --solver2=cvc5 --solverbin2=/home/cvc5/build/bin/cvc5

Since our tool is very easy to deploy, you can try to use it to find new bugs in solvers.

If you want to reproduce some bugs over the previous versions, you should use git checkout to transform the solvers to release versions or specific commits, which can be obtained from the corresponding bug reports.

 

Data

We also attach the code coverage data we obtained in our experiments to the zip file.

You can review them in  ./data.

Note that, once the paper is accepted, we will open source HistFuzz as well as the complete experimental data.

 

Bug reports

z3

Soundness

https://github.com/Z3Prover/z3/issues/6289 (fixed) 

https://github.com/Z3Prover/z3/issues/6167 (duplicate) 

https://github.com/Z3Prover/z3/issues/5798 (fixed) 

https://github.com/Z3Prover/z3/issues/5722#issue-1084089459 (fixed) 

https://github.com/Z3Prover/z3/issues/6116#issuecomment-1169467800 (fixed) 

https://github.com/Z3Prover/z3/issues/6079#issuecomment-1162563962 (reported) 

https://github.com/Z3Prover/z3/issues/6054#issuecomment-1138671939 (fixed) 

https://github.com/Z3Prover/z3/issues/6052 (fixed) 

https://github.com/Z3Prover/z3/issues/5939 (duplicate) 

https://github.com/Z3Prover/z3/issues/5932 (duplicate) 

https://github.com/Z3Prover/z3/issues/5851 (won't fix) 

https://github.com/Z3Prover/z3/issues/5777#issuecomment-1014065487 (fixed) 

https://github.com/Z3Prover/z3/issues/5753#issuecomment-1008214504 (won't fix) 

https://github.com/Z3Prover/z3/issues/5753#issuecomment-1007087883 (duplicate) 

https://github.com/Z3Prover/z3/issues/5737 (confirmed) 

https://github.com/Z3Prover/z3/issues/5722#issuecomment-998452982 (fixed) 

https://github.com/Z3Prover/z3/issues/4175#issuecomment-1189708587 (reported) 

Invalid Model

https://github.com/Z3Prover/z3/issues/6196#issuecomment-1204893767 (fixed) 

https://github.com/Z3Prover/z3/issues/6196 (fixed) 

https://github.com/Z3Prover/z3/issues/6165#issuecomment-1189714198 (fixed) 

https://github.com/Z3Prover/z3/issues/6165 (fixed) 

https://github.com/Z3Prover/z3/issues/6153 (fixed) 

https://github.com/Z3Prover/z3/issues/6116#issuecomment-1190944993 (fixed) 

https://github.com/Z3Prover/z3/issues/6113 (won't fix) 

https://github.com/Z3Prover/z3/issues/6105 (fixed) 

https://github.com/Z3Prover/z3/issues/6080 (reported) 

https://github.com/Z3Prover/z3/issues/6079 (reported) 

https://github.com/Z3Prover/z3/issues/6051 (reported) 

https://github.com/Z3Prover/z3/issues/5867 (fixed) 

https://github.com/Z3Prover/z3/issues/5826 (won't fix) 

https://github.com/Z3Prover/z3/issues/5825#issuecomment-1035013023 (fixed) 

https://github.com/Z3Prover/z3/issues/5825 (fixed) 

https://github.com/Z3Prover/z3/issues/5789 (fixed) 

https://github.com/Z3Prover/z3/issues/5778#issuecomment-1018132484 (fixed) 

https://github.com/Z3Prover/z3/issues/5769 (reported) 

Crash

https://github.com/Z3Prover/z3/issues/6271 (fixed) 

https://github.com/Z3Prover/z3/issues/6260 (fixed) 

https://github.com/Z3Prover/z3/issues/6177 (duplicate) 

https://github.com/Z3Prover/z3/issues/6174 (fixed) 

https://github.com/Z3Prover/z3/issues/6128 (fixed) 

https://github.com/Z3Prover/z3/issues/6116#issuecomment-1231402843 (fixed) 

https://github.com/Z3Prover/z3/issues/6116#issuecomment-1205949412 (fixed) 

https://github.com/Z3Prover/z3/issues/6116#issuecomment-1197613598 (fixed) 

https://github.com/Z3Prover/z3/issues/6116#issuecomment-1186831591 (reported) 

https://github.com/Z3Prover/z3/issues/6061 (fixed) 

https://github.com/Z3Prover/z3/issues/6060#issuecomment-1153378108 (fixed) 

https://github.com/Z3Prover/z3/issues/6060 (reported) 

https://github.com/Z3Prover/z3/issues/6058 (fixed) 

https://github.com/Z3Prover/z3/issues/5953 (fixed) 

https://github.com/Z3Prover/z3/issues/5942 (fixed) 

https://github.com/Z3Prover/z3/issues/5940 (won't fix) 

https://github.com/Z3Prover/z3/issues/5935 (fixed) 

https://github.com/Z3Prover/z3/issues/5928 (fixed) 

https://github.com/Z3Prover/z3/issues/5925 (confirmed) 

https://github.com/Z3Prover/z3/issues/5924 (won't fix) 

https://github.com/Z3Prover/z3/issues/5911 (fixed) 

https://github.com/Z3Prover/z3/issues/5857 (won't fix) 

https://github.com/Z3Prover/z3/issues/5852 (duplicate) 

https://github.com/Z3Prover/z3/issues/5816 (won't fix) 

https://github.com/Z3Prover/z3/issues/5804 (fixed) 

https://github.com/Z3Prover/z3/issues/5784 (fixed) 

https://github.com/Z3Prover/z3/issues/5779 (won't fix) 

https://github.com/Z3Prover/z3/issues/5778#issuecomment-1091354175 (fixed) 

https://github.com/Z3Prover/z3/issues/5778#issuecomment-1091259893 (fixed) 

https://github.com/Z3Prover/z3/issues/5778#issuecomment-1088211198 (fixed) 

https://github.com/Z3Prover/z3/issues/5778#issuecomment-1086766609 (fixed) 

https://github.com/Z3Prover/z3/issues/5778#issuecomment-1086765374 (won't fix) 

https://github.com/Z3Prover/z3/issues/5778#issuecomment-1086570683 (fixed) 

https://github.com/Z3Prover/z3/issues/5778#issuecomment-1086568308 (fixed) 

https://github.com/Z3Prover/z3/issues/5778#issuecomment-1079952765 (fixed) 

https://github.com/Z3Prover/z3/issues/5778#issuecomment-1079833134 (fixed) 

https://github.com/Z3Prover/z3/issues/5778#issuecomment-1077024464 (fixed) 

https://github.com/Z3Prover/z3/issues/5778#issuecomment-1074848129 (fixed) 

https://github.com/Z3Prover/z3/issues/5778#issuecomment-1074808615 (fixed) 

https://github.com/Z3Prover/z3/issues/5778#issuecomment-1044323782 (won't fix) 

https://github.com/Z3Prover/z3/issues/5778#issuecomment-1030572118 (fixed) 

https://github.com/Z3Prover/z3/issues/5778#issuecomment-1016018402 (fixed) 

https://github.com/Z3Prover/z3/issues/5770 (won't fix) 

https://github.com/Z3Prover/z3/issues/5758 (fixed) 

https://github.com/Z3Prover/z3/issues/5754 (duplicate) 

https://github.com/Z3Prover/z3/issues/5753#issuecomment-1012024921 (fixed) 

https://github.com/Z3Prover/z3/issues/5753#issuecomment-1011755733 (fixed) 

https://github.com/Z3Prover/z3/issues/5753#issuecomment-1011749964 (fixed) 

https://github.com/Z3Prover/z3/issues/5753#issuecomment-1011732992 (fixed) 

https://github.com/Z3Prover/z3/issues/5753#issuecomment-1007982562 (fixed) 

https://github.com/Z3Prover/z3/issues/5753#issuecomment-1007067681 (won't fix) 

https://github.com/Z3Prover/z3/issues/5741 (fixed) 

https://github.com/Z3Prover/z3/issues/5734 (fixed) 

https://github.com/Z3Prover/z3/issues/5732 (fixed) 

https://github.com/Z3Prover/z3/issues/5722#issuecomment-998652655 (won't fix) 

https://github.com/Z3Prover/z3/issues/5641#issuecomment-1000804719 (reported) 

 

cvc5

Soundness

https://github.com/cvc5/cvc5/issues/8572 (fixed) 

https://github.com/cvc5/cvc5/issues/8456 (fixed) 

https://github.com/cvc5/cvc5/issues/8052 (fixed) 

https://github.com/cvc5/cvc5/issues/8016 (fixed) 

https://github.com/cvc5/cvc5/issues/8890 (fixed) 

https://github.com/cvc5/cvc5/issues/7924 (fixed) 

Invalid Model

https://github.com/cvc5/cvc5/issues/8963 (fixed) 

https://github.com/cvc5/cvc5/issues/8916 (fixed) 

https://github.com/cvc5/cvc5/issues/8915 (fixed) 

https://github.com/cvc5/cvc5/issues/8515 (fixed) 

https://github.com/cvc5/cvc5/issues/8457 (fixed) 

https://github.com/cvc5/cvc5/issues/8412 (fixed) 

https://github.com/cvc5/cvc5/issues/8349 (reported) 

https://github.com/cvc5/cvc5/issues/8182 (fixed) 

https://github.com/cvc5/cvc5/issues/8160 (fixed) 

https://github.com/cvc5/cvc5/issues/8147 (fixed) 

https://github.com/cvc5/cvc5/issues/8133 (fixed) 

https://github.com/cvc5/cvc5/issues/8124 (fixed) 

https://github.com/cvc5/cvc5/issues/8118 (fixed) 

https://github.com/cvc5/cvc5/issues/7938 (fixed) 

Crash

https://github.com/cvc5/cvc5/issues/9040 (fixed) 

https://github.com/cvc5/cvc5/issues/8995 (fixed) 

https://github.com/cvc5/cvc5/issues/8987 (fixed) 

https://github.com/cvc5/cvc5/issues/8959 (fixed) 

https://github.com/cvc5/cvc5/issues/8954 (reported) 

https://github.com/cvc5/cvc5/issues/8912 (confirmed) 

https://github.com/cvc5/cvc5/issues/8906 (fixed) 

https://github.com/cvc5/cvc5/issues/8905 (fixed) 

https://github.com/cvc5/cvc5/issues/8895 (fixed) 

https://github.com/cvc5/cvc5/issues/8835 (fixed) 

https://github.com/cvc5/cvc5/issues/8773 (fixed) 

https://github.com/cvc5/cvc5/issues/8767 (reported) 

https://github.com/cvc5/cvc5/issues/8744 (fixed) 

https://github.com/cvc5/cvc5/issues/8609 (fixed) 

https://github.com/cvc5/cvc5/issues/8520 (fixed) 

https://github.com/cvc5/cvc5/issues/8517 (fixed) 

https://github.com/cvc5/cvc5/issues/8516 (fixed) 

https://github.com/cvc5/cvc5/issues/8411 (duplicate) 

https://github.com/cvc5/cvc5/issues/8410 (fixed) 

https://github.com/cvc5/cvc5/issues/8377 (fixed) 

https://github.com/cvc5/cvc5/issues/8348 (duplicate) 

https://github.com/cvc5/cvc5/issues/8183 (fixed) 

https://github.com/cvc5/cvc5/issues/8163 (fixed) 

https://github.com/cvc5/cvc5/issues/8162 (fixed) 

https://github.com/cvc5/cvc5/issues/8158 (fixed) 

https://github.com/cvc5/cvc5/issues/8119 (fixed) 

https://github.com/cvc5/cvc5/issues/8111 (fixed) 

https://github.com/cvc5/cvc5/issues/8094 (fixed) 

https://github.com/cvc5/cvc5/issues/8067 (fixed) 

https://github.com/cvc5/cvc5/issues/8001 (fixed) 

https://github.com/cvc5/cvc5/issues/7974 (fixed) 

https://github.com/cvc5/cvc5/issues/7937 (fixed) 

https://github.com/cvc5/cvc5/issues/7919 (duplicate) 

https://github.com/cvc5/cvc5/issues/7918 (fixed) 

https://github.com/cvc5/cvc5/issues/7917 (fixed) 

https://github.com/cvc5/cvc5/issues/7902 (fixed) 

https://github.com/cvc5/cvc5/issues/7894 (fixed) 

https://github.com/cvc5/cvc5/issues/7893 (fixed) 

https://github.com/cvc5/cvc5/issues/7858 (duplicate)