I. Very quick guide
1. What to run?
python reproduce.py
2. What results to expect?
- all results are displayed on the terminal output
- the following files are generated:
- for the tool paper examples:
models/result-A-1.res, …,models/result-A-9.res - for the tool paper figures:
models/result-A-10-statespace.pdf(Fig 1a) andmodels/result-A-11_cart.png(Fig 1b) - for the additional experiments:
models/result-B-1.restomodels/result-B-5.res
- for the tool paper examples:
3. What execution time to expect?
Probably less than one minute, including at most a few seconds for the first part (tool paper examples).
The rest of this file contains more detailed instructions.
II. What does the artifact contain?
1. The necessary models and commands to reproduce all executions mentioned in the submitted paper
The tool paper uses as a motivating example an ad-hoc model of a researcher writing papers and drinking coffee. This is a toy example specifically designed to exemplify the algorithms offered by IMITATOR 3; however, it is directly inspired by some existing benchmarks (such as producer-consumer and scheduling examples).
The tool paper mentions 9 different properties, that are all reproduced automatically.
2. An additional set of experiments to demonstrate IMITATOR capabilities in various applications domains
These case studies were selected as follows:
- they answer meaningful problems, mostly from actual industrial collaborations
- they are relatively fast to execute (less than 1 minute each)
- they give an idea of the wide applicability of IMITATOR to various research domains, and of its efficiency in managing non-trivial benchmarks in a reasonable execution time
Verification of the SPSMALL memory circuit (collaboration with ST-microelectronics)
Source: no exact published source, but model close to the one from: Chevallier, R., Encrenaz-Tiphène, E., Fribourg, L., Xu, W.: Timed verification of the generic architecture of a memory circuit using parametric timed automata. FMSD 34(1), 59–81 (Feb 2009). 10.1007/s10703-008-0061-x
Objective: derive conditions over the input and internal timing parameters for which the functional and timed behavior is guaranteed
Result: condition for which the functional and timed behavior is guaranteed
Verifying the multi-processor image processing system of an unmanned aerial aircraft with uncertain periods, as a benchmark made public by Thales
Source: Étienne André, Giuseppe Lipari and Sun Youcheng. IMITATOR: Formal Verification of Real-Time Systems Under Uncertainty. In Steve Goddard and Harini Ramaprasad (eds.), ECRTS’15, July 2015.
Objective: derive the minimal and maximal processing time, in presence of uncertainties and possible frame losses
Expected result: The end-to-end time is between 63 and 145.008
(output by IMITATOR in the form 18126 >= 125*e2e & e2e >= 63)
Parametric pattern matching and monitoring of logs from the automative industry
Source: Étienne André, Ichiro Hasuo and Masaki Waga. Offline timed pattern matching under uncertainty. In Anthony Widjaja Lin and Jun Sun (eds.), ICECCS’18, IEEE CPS, pages 10–20, December 2018. 10.1109/ICECCS2018.2018.00010
Objective: derive start and end time over the log, plus parameter valuations, for which the specification is satisfied
Parametric opacity analyses for timed automata
Source: Étienne André and Sun Jun. Parametric timed model checking for guaranteeing timed opacity. In Yu-Fang Chen, Chih-Hong Cheng and Javier Esparza (eds.), ATVA’19, Springer LNCS 11781, pages 115–130, October 2019. 10.1007/978-3-030-31784-3_7
Objective: derive parameter valuations guaranteeing that there is no timing leak to an observer who can access the computation time of a Java program
Verification of the Bounded Retransmission Protocol (BRP)
Source: Étienne André, Jaime Arias, Laure Petrucci and Jaco van de Pol. Iterative bounded synthesis for efficient cycle detection in parametric timed automata. In Jan Friso Groote and Kim G. Larsen (eds.), TACAS’21, Springer LNCS 12651, pages 311-329, 10.1007/978-3-030-72016-2_17 This version of the BRP model was written by Laure Petrucci and Jaco van de Pol
Objective: derive conditions for which the channels will never be used simultaneously
Expected result: note the constraint TS > 2*TD, which is the "exact" expected result
III. How to run the artifact within the virtual machine?
1. Virtual machine details
The virtual machine OS is Linux Mint 20.1 Ulyssa Xfce edition 64 bits
User : imitator
Password: IMITATOR
2. Which command to execute?
cd /home/imitator/artifact/
python reproduce.py
The reproduce.py script runs IMITATOR for a number of
models and properties, displays the outputs (i.e., the synthesized
constraints) onto the terminal, and generates a number of files.
3. Where to get the results?
- All results are displayed on the terminal output
- In addition, the following files are generated:
- Results for the properties given in the tool paper:
models/result-A-1.res, …,models/result-A-9.res - Figures from the tool paper regenerated:
models/result-A-10-statespace.pdf(Fig 1a) andmodels/result-A-11_cart.png(Fig 1b) - Additional experiments results:
models/result-B-1.restomodels/result-B-5.res
- Results for the properties given in the tool paper:
IV. How to reproduce the results outside the virtual machine?
It is possible to run the experiments outside of the provided virtual machine, i.e., simply from the binary and the models.
IMITATOR is distributed in the form of a standalone binary (without dependencies).
Therefore, the artifact can be reproduced on any Linux-based computer (64 bits), with Python (required to run the reproducibility script).
That is, simply download IMITATOR 3 and run the artifact as specified above.
Only if you want to reproduce graphics, you may have to install dot and graphviz (packages plotutils and graphviz on Debian).
V. How to reproduce by installing everything from the source?
IMITATOR can be compiled without too much effort on any Linux-based computer (64 bits).
A quick guide is reproduced below:
1. Install the following packages
sudo apt-get install python g++ m4 ocaml libextlib-ocaml libextlib-ocaml-dev libfileutils-ocaml-dev ocaml-findlib libgmp-dev libgmp-ocaml libgmp-ocaml-dev libppl-dev openmpi-bin openmpi-common libopenmpi-dev git oasis graphviz plotutils
(Installing these libraries might entail additional dependencies.)
2. Download the PPL source
Directly on their Website
3. Compile the PPL binding for OCaml
cd ppl-1.2
./configure --prefix=/usr
cd interfaces/OCaml/
make
sudo make install
The make operation can take some time (a number of minutes).
4. Download the IMITATOR source
git clone https://github.com/imitator-model-checker/imitator.git
Or, to get exactly the v3.0.0 version, download and unzip:
https://github.com/imitator-model-checker/imitator/archive/refs/tags/v3.0.0.tar.gz
5. Copy the METAS files
cd imitator
sudo cp METAS/* /usr/lib/ocaml/METAS/
6. Compile IMITATOR
sh build.sh
Enjoy!
A fully detailed guide for installing IMITATOR is available online.
Also note that a Docker version of IMITATOR is available:
docker pull imitator/imitator
VI. How to reproduce further results?
The user manual provides both a quick and a very detailed description of the syntax, and how to run analyses.
Overall, a model can be written by following the syntax, and the usual call to IMITATOR is as follows:
./imitator-v3.0.0-amd64 model.imi property.imiprop
We also provide users with various benchmarks:
Example of an additional experiment that can be run
We give above a number of locations to get IMITATOR benchmarks. As an example of input beyond those described above, let us consider the Fischer mutual exclusion protocol for 3 processes.
A model can be downloaded in the v3.0.0 repository here:
https://raw.githubusercontent.com/imitator-model-checker/imitator/v3.0.0/benchmarks/Fischer/F3.imi
The safety property can be downloaded in the v3.0.0 repository here:
The analysis command is simply
./imitator-v3.0.0-amd64 F3.imi Fi.imiprop
The constraint guaranteeing the protocol safety ("no more than 1 process at once in the critical section") is:
delta >= 0 & Delta >= delta
VII. Miscellaneous
1. Missing experiment
In the paper, we mentioned that an experiment does not terminate (i.e., the parameter deadlock freeness synthesis). For sake of simplicity, we did not include it in the reproducibility script (obviously because it does not terminate). It can be manually executed as follows:
./imitator-v3.0.0-amd64 models/researcher.imi models/researcher-deadlockfree.imiprop
Note that, as mentioned in the paper, by setting an exploration depth limit (option -depth-limit 40), the synthesis terminates; this latter analysis is included in our reproducibility script.
2. Result printing discrepancy
While all algorithms in IMITATOR are (by default) exact, the implementation within IMITATOR of the minimal-time reachability algorithm encodes strict constraints with an approximation, i.e., by adding or substracting a (constant) epsilon.
That is, the verbatim result of the minimal-time reachability property is displayed by IMITATOR as:
… & 2000100000000001 >= 500000000000000*pTotal + 1000000000000000*pNeed + 2000000000000000*pCoffee
which should be read as:
20 >= 5*pTotal + 10*pNeed + 20*pCoffee
or 2 >= 1/2*pTotal + pNeed + 2*pCoffee (as in the paper)
This (manual) simplification was not made explicit in the tool paper submission (and probably should have). Fixing the implementation to replace this approximation with the exact behavior of other algorithms (such as minimal-parameter reachability) is on our agenda for the next release.
The minimal- or maximal-parameter reachability algorithms are not concerned by this phenomenon.
VIII. Contact
This artifact was made by Étienne André.
Official Web site: imitator.fr