Artifact for "Explainable Online Monitoring of Metric Temporal Logic"

This artifact supplements our paper Explainable Online Monitoring of Metric Temporal Logic, which was accepted for publication and presentation at TACAS 2023. It includes all dependencies, tools and instructions to locally execute our experiments using the TACAS 2023 Artifact Evaluation Virtual Machine for VirtualBox. In particular, we describe the steps to setup a differential testing environment and empirically evaluate our tool by comparing it to other state-of-the-art monitoring tools.

Reproducibility Requirements

We assume that Explanator2 (i.e., the root folder) is located at /home/tacas23/explanator2. 

First, install all Debian packages with

$ cd packages/deb/
$ sudo dpkg -i *.deb

Note that some packages might not succeed (e.g., python3-clang), but you can still proceed with the next steps. Subsequently, install all pip3 packages with

$ cd packages/whl/
$ find . -maxdepth 1 -type f -exec pip3 install {} \; > /dev/null

Lastly, setup opam and install all related packages with

$ opam init local /home/tacas23/explanator2/packages/opam-repository -y --disable-sandboxing
$ eval $(opam env --switch=default)
$ opam install -y ocamlbuild ocamlfind ctypes dune dune-build-info safa menhir qcheck zarith core_kernel base js_of_ocaml js_of_ocaml-ppx js_of_ocaml-lwt zarith_stubs_js
$ eval $(opam env)

Differential Testing

If you want to test Explanator2, first run

$ dune build

from inside the root folder. Next, run

$ ./run_tests.sh [number of seeds]

from inside the test folder. Also from inside the test folder, you can remove the temporary folder with

$ ./clean.sh

Evaluation

These are the basic steps to take if you want to evaluate Explanator2.

First, compile Explanator2, Aerial, MonPoly, Verimon, Hydra and Vydra with

$ cd hydra
$ ./pre.sh

Next, you can run the empirical evaluation included in the paper by running

$ ./run_exp.sh config_paper.py

from inside the hydra folder. After this script successfully finishes, the data from the experiments will be stored in hydra/evaluation/stats, and the plots (corresponding to Fig. 12) will be stored in hydra/evaluation/figs.

README.md contains instructions and further information.