Octopus

Artifacts for "Oracle-Guided SMT Solver Validation via Synthesizing Diverse Formulas with Multiple Assignments"

Table of Contents

• Background
• Prerequisites
• Usage
• Experimental Reproduction

Background

Octopus is a cutting-edge tool for detecting bugs in Satisfiability Modulo Theories (SMT) solvers.

By rigorously testing and validating these solvers, Octopus has successfully reported 13 bugs across popular SMT solvers like Z3, cvc5, and Bitwuzla. Among these, 9 bugs have been confirmed and subsequently fixed by the developers, demonstrating the tool's effectiveness and reliability.

Install

Octopus has minimal dependencies, making it easy to set up and use. You'll need the following:

• Python 3.x: Required for executing Octopus.

• Python Packages: z3-solver, termcolor, configparser, antlr4-python3-runtime

  To install these packages, run:

pip3 install z3-solver termcolor configparser antlr4-python3-runtime

Usage

• Downloading Benchmarks

To work with SMT-LIB official benchmarks, you can use the provided bash script. Before running the script, make sure to uncomment the lines corresponding to the logic you want to test.

• Setting Up the SMT Solver

To validate an SMT solver, you'll first need to install and build it. Here's how to set up Z3:

1. Clone the Z3 Repository:

git clone https://github.com/Z3Prover/z3.git

2. Build Z3

python3 scripts/mk_make.py

cd build

make

• Running Octopus

After setting up the benchmarks and the SMT solver, you can use Octopus for validation. Below are examples of how to run Octopus:

1. Basic Validation Command:

python3 __main__.py --benchmark=/path/to/benchmarks --solver=z3 --solverbin=/path/to/z3/build/z3 --theory=LIA

2. Using Multiple Cores: To expedite the validation process, Octopus supports multi-core processing:

python3 __main__.py --benchmark=/path/to/benchmarks --solver=z3 --solverbin=/path/to/z3/build/z3 --theory=LIA --cores=20

Experimental Reproduction

To compare Octopus with other state-of-the-art fuzzers, you’ll need to install the necessary tools:

1, Install the required Python packages:

pip3 install efficient-apriori github3.py

2, For installing STORM, refer to the installation instructions.

• Code Coverage

To replicate the code coverage experiments conducted with different fuzzers, you need to enable gcov during the build process of the solvers. For example, to build Z3 with gcov support, follow these commands:

git clone https://github.com/Z3Prover/z3.git
cd z3
CFLAGS=--coverage CXXFLAGS=--coverage LDFLAGS=--coverage python3 scripts/mk_make.py
cd build
make

For building cvc5 with gcov enabled, refer to the official documentation.

For more details, please refer to the README.md file.