Building / Installing

You have three options:

1. Load the provided image docker load -i image.txz
2. Run docker build -t mdp-reach-avoid . to build the docker image
3. Manual setup:
   • Ensure that your Python version can create virtual environments
   • Run python -m venv venv to create such a venv
   • Activate it with . ./venv/bin/activate
   • pip install -r requirements.txt to install dependencies
   • (optional, but recommended) Download Yices from https://yices.csl.sri.com/ and extract yices-version/bin/yices-smt2 (note the -smt2!) into the directory.

Running

(If you are using the docker image, run docker run -it mdp-reach-avoid to switch to the image.)

(You may need to run python3 instead of python depending on your distribution.)

Simply run python main.py --experiment <name> --method <method> to run the respective experiment. Available experiments are: running, slippery, two_init, grid_5_4, grid_8_8, grid_20_10, insulin, page_rank. Specify --method synth or --method cert to run either synthesis or certification. Together, a complete execution would look like python main.py --experiment running --method synth to run policy synthesis on the running example.

Run python main.py --help to view help and add --debug to view additional information (such as the generated SMT query, the response of the solver, etc.)

Note that due to randomness in initialization, it can somethimes take much longer to find a solution than specified in the paper. If that is the case, simply abort and restart the process.

Code Structure

• main.py contains the overall organization of the code
• implementation/algorithm.py contains the high-level algorithm, i.e. generating the templates and interpreting the results of the SMT solver
• implementation/constraints.py contains constraint generation and quantifier elimination
• implementation/config.py just contains a few utility classes
• implementation/model.py contains classes related to representing MDP
• implementation/experiments.py contains the actual list of experiments