# ARTIFACT OF "Proving the existence of fair paths in infinite state systems"

This document describes how to use the artifact to reproduce the
results shown in the experimental evaluation of the paper titled
`Proving the existence of fair paths in infinite state systems`
submitted to VMCAI 2021.

The commands reported in this README are preceded by the $> prompt.

The artifact includes:
  * 4 bash scripts that install the required tools in the home
    directory;
  * the scripts required to re-run the experimental evaluation shown
    in the paper;
  * the scripts required to generate the plots;
  * the output files from which the plots included in the paper have
    been generated.



## STRUCTURE OF THE ARTIFACT

In the following we briefly explain the content of the directories
included in the artifact.
  * `anant`                 includes the source code of the tool Anant
                            and a helper bash script used to run it.

  * `anant_pip_packages`    includes the python packages required by
                            Anant.

  * `AProVE`                includes the jar archive required to run
                            AProVE, some binary files required by the
                            tool and the source code  of its
                            dependecies.

  * `aprove_apt_debs`       includes the deb packages required to
                            compile and run AProVE and its
                            dependencies.

  * `benchmarks`            includes the input files for Anant, AProVE
                            and nuXmv. These are the tests used to compare FairFind with these
                            tools.

  * `FairFind`              contains the source code and benchmarks
                            for FairFind.

  * `FairFind_apt_debs`     contains the deb packages required to run
                            FairFind.

  * `FairFind_pip_packages` contains the python dependencies required
                            to run FairFind.

  * `pysmt-solvers`         includes the archives required to install
                            the SMT solvers mathsat and z3 for the python package pysmt, used
                            by FairFind.

  * `pysmt`                 includes the source code of the pysmt
                            python library.

  * `common-debs`           includes the packages to install runlim,
                            python-pip and python-venv.



## INSTALLING THE TOOLS

This section reports the commands required to install the tools.
The tools are installed in the $HOME directory.

In the following let $ROOT be the directory `artifact`, created in
extracting the zip archive.
If the current directory is the root of the extracted zip archive, the
following command sets ROOT.

  $> export ROOT=$(pwd)/artifact

First run `install_common.sh` to install python-pip, venv and binary,
then reload the `.bashrc`.

  $> $ROOT/install_common.sh
  $> source $HOME/.bashrc


### FAIR FIND
The script `install_fair-find.sh` installs FairFind and its
dependencies in `$HOME/FairFind`.
When installing the SMT-solvers mathsat and z3 the script will ask you
to accept their licences, please do by typing `Y` and enter.

  $> $ROOT/install_fair-find.sh


### ANANT
The script `install_anant.sh` installs Anant and its dependencies in
`$HOME/anant`.

  $> $ROOT/install_anant.sh


### AProVE
The script `install_aprove.sh` installs AProVE and its dependencies in
`$HOME/AProVE`.
The artifact contains a pre-compiled version, suitable for the
VMCAI-2021 ubuntu20 virtual machine, of the dependencies of AProVE.
If you need to recompile them please uncomment the related sections in
the `install_aprove.sh` script. Notice that recompiling them could
take some minutes.

  $> $ROOT/install_aprove.sh



## CHECKING THE INSTALLATION

The following commands can be used to quickly check if the
installation has been successful and whether the script for generating
the plots works.

First activate the python virtual environment created for FairFind.

  $> source $HOME/FairFind/venv/bin/activate


From this virtualenv it is possible to run FairFind on a single
  benchmark by specifying the $TEST_FILE. $TEST_FILE is the path of
  one of the "*.py" files contained in the directories `tests`,
  `scaled_bench_19`, `scaled_example2` or `scaled_bouncing_ball`,
  placed in $HOME/FairFind. If you use one of the scaling benchmarks,
  please note that the number at the beginning of the name is the
  number of AG-skeletons in that example: the higher the number the
  higher the execution time.

  $> python3 $HOME/FairFind/src/run_tests.py --nuxmv $HOME/.local/bin/nuXmv --output /tmp $TEST_FILE


The following command, instead, executes all benchmarks used to
  compare FairFind with other tools. On our 8 GB, Intel i5 machine it
  took about 10 minutes.

  $> python3 $HOME/FairFind/src/run_tests.py --nuxmv $HOME/.local/bin/nuXmv --output /tmp $HOME/FairFind/tests


In both cases FairFind should run without any errors. After each test
you should see on the standard output "end of `<test-name>`: SUCCESS",
where <test-name> is the name of the test.
  * SUCCESS means that FairFind has been able to identify a R-abstraction,
  * FAILURE is printed when FairFind is unable to identify a R-abstraction,
  * UNKNOWN is printed when FairFind identified some candidate
            composition, but is unable to determine whether such composition
            satisfies the compatibility constraint.


Finally you can run `create_plots.py` to obtain the plots included in
the paper (figures 3 and 4).
First activate the FairFind virtual environment and then run
`create_plots.py`.

  $> source $HOME/FairFind/venv/bin/activate
  $> python3 $HOME/FairFind/plots/create_plots.py --aprove_out_dir $HOME/paper_data/figure3/aprove --anant_out_dir $HOME/paper_data/figure3/anant --nuxmv_out_dir $HOME/paper_data/figure3/nuxmv --fairfind_out_file $HOME/paper_data/figure3/fairfind.out --scaling_bouncing_ball_out_file $HOME/paper_data/figure4/scaling_bouncing_ball.out --scaling_bench19_out_file $HOME/paper_data/figure4/scaling_bench19.out --scaling_example2_out_file $HOME/paper_data/figure4/scaling_example2.out



## RUNNING THE EXPERIMENTS

This section first describes how to run each tool to reproduce the
experiments reported in Figure 3, and then how to run FairFind on the
scaling benchmarks and obtain the results reported in Figure 4.

We run the experiments on a Ubuntu 18.04 machine with 8 GB of memory
and processor Intel(R) Core(TM) i5-3230M CPU @ 2.60GHz.

Please, consider that running the experiments on a machine with lower
resources might cause additional timeouts in the SMT-queries with
respect to the ones we obtained in our experiments. Additional
timeouts could result in the generation of a reachability problem with
less information and nuXmv might identify a different candidate
composition or, if some essential information is missing, the
reachability problem could have no solution.
If this is the case and you obtain some failures in the synthesis of
the R-abstraction, please consider setting a higher timeout for the
different steps of FairFind (see section on command line options).

We successfully re-executed FairFind on all benchmarks using the
Ubuntu20 virtual machine provided for the VMCAI2021 conference using
VirtualBox6.1 and assigning 3 CPUs and 6144 MB of memory, on our  8
GB, Intel i5 machine.



### COMPARISON WITH ANANT, APROVE AND NUXMV (Figure 3)

The data from these benchmarks has been used to generate Figure 3.

#### RUN ANANT
The bash script `run_anant.sh` runs anant on each of the benchmarks
and stores the outputs in a directory called "output_" followed by the
current date and hour in $HOME/benchmarks/anant.
The script by default uses a timeout of 10 minutes for each benchmark,
this can be changed by editing the script.
First activate the python virtual environment and then, run the
experiments by executing the bash script.
This script required 15 minutes to complete.

  $> source $HOME/anant/venv/bin/activate
  $> $HOME/benchmarks/anant/run_anant.sh


### RUN APROVE
The bash script `run_aprove.sh` runs AProVE on each of the benchmarks
and stores the outputs in a directory called "output_" followed by the
current date and hour in $HOME/benchmarks/aprove.
The script by default uses a timeout of 10 minutes for each benchmark,
this can be changed by editing the script.
This script required 3 hours to complete.

  $> $HOME/benchmarks/aprove/run_aprove.sh


### RUN NUXMV
The bash script `run_nuxmv.sh` runs AProVE on each of the benchmarks
and stores the outputs in a directory called "output_" followed by the
current date and hour in $HOME/benchmarks/smv.
The script by default uses a timeout of 10 minutes for each benchmark,
this can be changed by editing the script.
This script required 5 hours to complete.

  $> $HOME/benchmarks/smv/run_nuxmv.sh


### RUN FAIRFIND
The script `run_tests.py` is the main script that can be used to run
FairFind on the benchmarks considered in the paper.
First it is necessary to activate the python virtual environment.
Then, run FairFind on the benchmarks used for the comparison with
Anant, AProVE and nuXmv.
This script required 10 minutes to complete.
The command saves the output on a file `tests.out`, this file can be
later used as input to the script `create_plots.py` to generate the plots.

  $> source $HOME/FairFind/venv/bin/activate
  $> python3 $HOME/FairFind/src/run_tests.py --nuxmv $HOME/.local/bin/nuXmv --output /tmp $HOME/FairFind/tests > tests.out



### RUNNING FAIRFIND ON THE SCALING BENCHMARKS (Figure 4)

The data from this benchmarks has been used to generate Figure 4.

Notice that some examples require a large amount of memory. In our
experiments 6 GB have been enough to complete all tests.
If you have a limited amount of memory please consider using `ulimit
-Sm 4000000`, before running the commands below, to kill the execution
of the tests when the usage of memory exceeds 4 GB.

The following commands allow the execution of FairFind on the three
different scaling benchmarks: bouncing-ball, bench-19 and example-2.
The output is stored on a file that can be used as input to the
`create_plots.py` script to generate the plot reported in Figure 4.

  $> python3 $HOME/FairFind/src/run_tests.py --nuxmv $HOME/.local/bin/nuXmv --output /tmp $HOME/FairFind/scaled_bench_19 > scaling_bench19.out
  $> python3 $HOME/FairFind/src/run_tests.py --nuxmv $HOME/.local/bin/nuXmv --output /tmp $HOME/FairFind/scaled_example2 > scaling_example2.out
  $> python3 $HOME/FairFind/src/run_tests.py --nuxmv $HOME/.local/bin/nuXmv --output /tmp $HOME/FairFind/scaled_bouncing_ball > scaling_bouncing_ball.out

The first command, on our machine, took about 10 minutes.
The second command, on our machine, took about 2 hours.
The third command, on our machine, took about 5 hours.



## COMMAND LINE OPTIONS TO FAIRFIND

In the following we provide more details about the command line
options available for `run_tests.py`.
  * --help               : prints a list and brief description of the
                           available command line options.

  * --nuxmv              : specifies the path of the nuXmv binary to
                           be used.

  * --output             : specifies a directory where the input
                           reachability problem for nuXmv, the
                           commands for nuXmv, the witness generated
                           by nuXmv and the SMV model representing the
                           final R-abstraction are stored. By default
                           these files are deleted at the end of the
                           test, see `--no-clear`.

  * --no-clear           : do not delete temporary files upon
                           completion.

  * --smv-comp-timeout   : sets the timeout in seconds for each
                           SMT-query performed in the computation of
                           the encoding of the search for the
                           composition as a reachability problem.

  * --check-comp-timeout : sets the timeout in seconds for each
                           SMT-query performed for the additional
                           check of correctness performed on the final result.

  * --find-comp-timeout  : sets the timeout in seconds for each
                           SMT-query performed to check whether a
                           candidate composition satisfies the
                           assume-guarantee constraints: transitions
                           are compatible.

  * --nuxmv-timeout      : sets the timeout in seconds for each run of
                           nuXmv to find a candidate composition.

  * --verbose            : enable verbose printing.

  * --try-sync-product   : after computing the subset of hints
                           sufficient to obtain a valid composition,
                           check whether the transition relation of
                           the naive synchronous product of such
                           transition systems implies the transition
                           relation of the original system. This
                           highlights the significance of the search
                           for the combination of the regions to be
                           composed, instead of just picking the
                           hints.

As final argument the script requires a list on tests to be
executed. If the input is a single test file, such test is executed,
if the input is a directory the script assumes that every python
script inside the directory is a test and executes all of them.



## CREATING THE PLOTS

The script `plots/create_plots.py` generates the plots included in the
paper and also some additional ones.
Please activate the python environment created for FairFind:

  $> source $HOME/FairFind/venv/bin/activate


Then the following command generates the plots:

  $> python3 $HOME/plots/create_plots.py --aprove_out_dir $APROVE_OUT_DIR --anant_out_dir $ANANT_OUT_DIR --nuxmv_out_dir $NUXMV_OUT_DIR --fairfind_out_file $FAIRFIND_OUT_FILE --scaling_bouncing_ball_out_file $SCALING_BOUNCING_BALL --scaling_bench19_out_file $SCALING_BENCH19 --scaling_example2_out_file $SCALING_EXAMPLE2

where:
  * $APROVE_OUT_DIR is the directory where the output files of AProVE
                    are stored,

  * $ANANT_OUT_DIR is the directory where the output files of Anant
                   are stored,

  * $NUXMV_OUT_DIR is the directory where the output files of nuXmv
                   are stored,

  * $FAIRFIND_OUT_FILE is the output file of FairFind run on the
                       benchmarks in the directory
                       `$HOME/FairFind/tests`,

  * $SCALING_BOUNCING_BALL is the output file of FairFind run on the
                           scaling bouncing ball benchmarks.
                           This parameter is optional.

  * $SCALING_BENCH19 is the output file of FairFind run on the scaling
                     bench_19 benchmarks.
                     This parameter is optional.

  * $SCALING_EXAMPLE2 is the output file of FairFind run on the
                      scaling example2 benchmarks.
                      This parameter is optional.


The following command generates the plots from the data we used to
generate the ones included in the paper.

    $> python3 $HOME/FairFind/plots/create_plots.py --aprove_out_dir $HOME/paper_data/figure3/aprove --anant_out_dir $HOME/paper_data/figure3/anant --nuxmv_out_dir $HOME/paper_data/figure3/nuxmv --fairfind_out_file $HOME/paper_data/figure3/fairfind.out --scaling_bouncing_ball_out_file $HOME/paper_data/figure4/scaling_bouncing_ball.out --scaling_bench19_out_file $HOME/paper_data/figure4/scaling_bench19.out --scaling_example2_out_file $HOME/paper_data/figure4/scaling_example2.out
