Reproducibility Artifacts for Bridging the Gap between PDES and Agent-Based Modeling and Simulation

PADS 2019 - Reproducibility

We thank the reviewer for the time and effort that they will dedicate to the reproducibility of the results contained in the paper submitted to PADS 2019 titled "Bridging the Gap between PDES and Agent-Based Modeling and Simulation".

We have opted in the reproducibility initiative asking for the following badges:

• Artifact evaluated functional/reusable

• Artifact available

• Results replicated

The proposal has been implemented within ROOT-Sim. The source code contained in this archive refers to the branch abm_layer on the official github repository, which will be soon merged into master, accompanying the upcoming new release of the runtime environment.

All experimental tests have been run on a distributed environment, composed of 3 heterogeneous machines, for a total of 112 cores. The amount of available RAM on the cluster was 192 GB. While we expect the results to be similar on different environments, there could be some need to fine-tune the runtime environemnt with respect to different hardware environments used in the reproducibility phase. We have taken all the possible actions to simplify at most this phase, reducing the effort demanded from the reviewer. Anyhow, if any problem shall arise, we are available to try to solve together possible issues---just drop us an email!

In the following sections, we describe why we have asked for the three specific badges, and why we consider our artifact suitable for each badge. Moreover, we provide instructions to carry out the replication of the results presented in the original submitted paper. The paper is also contained in this archive, for simpler reference.

Thanks again for the time spent in this process---we really appreciate it, and we understand the complexity associated with running a distributed runtime environment implemented by other people.

Errata and Shepherding

There has been a material error in the version of the paper which has been submitted for review. In particular, the script to generate the plot reported in Figure 6 of the original paper was referencing a wrong data file. This is clear when reading the comment to the figure, which reads in the original paper as:

For the sake of completeness, in Figure 6 we report performance data taken when running on the largest single node of our cluster, fo the real-world applications. Here, we do not pay communication costs, but the trends of the curves are perfectly similar to the ones shown in Figure 5, thus showing that the proposed API and its reference implementation is also resilient to network delays. 

By looking at the figure in the original submitted paper, two points are clear:

• The plot is reporting also data associated with a synthetic benchmark;

• The trends of one of the two curves is not comparable with those in Figure 5.

This material error arose from the fact that we had collected many more results during the experimentation phase, with respect to those published in the paper. The selection was carried out for the sake of space in the paper. Figure 6 in the original paper was plotted by using wrong data from a different experiment, not included in the final paper.

In the PDF paper included in this archive we have corrected Figure 6, as it should have appeared in the version submitted for review. We feel sorry for this problem, and we hope that this change allows the reproducibility reviewer to carry out their tasks in a suitable way anyhow.

Of course, the corrected Figure will appear in the camera ready version of the paper.

As an additional note, the paper has been conditionally accepted at the conference. The requests raised by the reviewers will not likely affect the experimental results in the paper, so the reproducibility can be carried as if the paper had not been shepherded.

Artifacts Evaluated Functional/Reusable

In the runtime subfolder of this archive the whole source code aligned to the latest commit in the abm_layer branch can be found. With respect to the results in the paper, the runtime folder contains the whole ROOT-Sim library and the models (even more models than those used in the experimental assessment in the paper).

There are therefore two different places to look at:

• The implementation of the ABM layer described in the paper can be found in runtime/src/lib/abm_layer.{c,h}. In particular, all API functions described in the paper are implemented in the C module. There is an additional header file in which API functions are exposed (and some data structure definitions are placed) which is located in runtime/src/ROOT-Sim.h. These three files contain all the API functions described in the paper, as a reference implementation.

• Simulation models used for the experimentation are located in runtime/models/. There is one subfolder for each simulation model. The ones which have been used for the experimentation are tcar_abm, tuberculosis, stupid, segregation_abm.

The reviewer can point to these files to check whether the implementation is compliant with the description in the paper, both for models and for the API support. We also encourage the reviewer to assess the level of documentation provided in the runtime environment code.

For the sake of reusability, ROOT-Sim (and the included ABM API) is a general-purpose runtime environment. The authors have spent a significant time at documenting the whole project and the ABM API. While the documentation is already published online in the project wiki (the AMB API is documented specifically here, and the topology library is documented here), there is also an ongoing effort at providing a complete documentation also of the internals. ROOT-Sim is a complex runtime environment, and finishing the documentation of the internals will necessarily take more time, as this is something which is not strictly related only to this publication.

The current version of the (online) internals documentation can be found here and here. The documentation contains all the information required to compile and run the runtime environment, with different modules.

We have anyhow integrated in this archive also two different PDF versions of the current ROOT-Sim documentation, which includes also the documentation of the usage and of the internals of the results presented in this paper. Documentation is located in the doc subfolder.

We hope the reviewer will see the effort which we are spending at documenting the whole ROOT-Sim package, and will find this documentation in a sufficiently mature state to assign the Reusable badge.

As for the other points associated with the artifact functional/reusable badge, we refer the reviewer to the "Results replicated" section , in which we describe how to exercise and run the environment. The description of the steps to take for re-running the experiments is tailored to the results in the paper, but as it will be shown it is general enough to be applied to any simulation model implemented respecting the ROOT-Sim API.

Artifacts Available

ROOT-Sim is an open source package. It's source code is available on the GitHub Repository, where the most part of the development process is taking place. Moreover, the project is supported by the official website.

Nevertheless, to make persistent the availability of the artifacts associated with the paper presented to PADS 2019, the content of this archive has also been uploaded to Zenodo. Zenodo respects all the requirements of durability and persistency, in particular:

• A unique identifier (DOI) is provided;

• There are explicit policies for disaster recovery;

• There are explicit policies for archive retention and longevity.

The whole Zenodo policy can be found here.

The DOI assigned to this artifact is: 10.5281/zenodo.2597110

Results Replicated

In order to compile ROOT-Sim, the following dependencies must be met:

• A modern version of gcc supporting the C11 standard (at least version 6 is suggested, so as to have efficient and not buggy implementations of the C11 standard)

• An MPI runtime environment.

As for the latter point, ROOT-Sim relies on modern MPI-3 facilities, in particular asynchronous multithreaded implementations of functions for message passing and collective operations. While the MPI-3 standard is not recent, common runtime environments have implemented in a sufficiently stable way these MPI primitives only recently.

In particular, we know that OpenMPI < 3.0.1 is known to be buggy in the implementation of some primitives which we use in ROOT-Sim, while MPICH < 3.2.0 has shown runtime problems under heavy loads in some of our experimentations. If you rely on non-compliant versions, there could be deadlocks or crashes while computing the GVT reduction. If you rely on other MPI implementations, there could also be problems there, which we might be unaware of.

We also make use of some modern Linux services in ROOT-Sim. A version of Linux >= 2.6.5 is required to compile and run ROOT-Sim models.

In order to compile ROOT-Sim, the following dependencies are required:

• autoconf

• automake

• libtoolize

• make

They should be anyhow available on any modern Linux distribution.

We hereby describe the steps to compile and install ROOT-Sim in your environment. For the sake of simplicity, we provide the script compile-and-install.sh which should automatize these steps.

We provide scripts to automatically re-run the tests. These scripts will generate figures, numbered as in the original paper. To support the automatic generation of these results, the following is also needed:

• a python3 interpreter

• gnuplot

Compiling and Installing ROOT-Sim

If you do not want to rely on compile-and-install.sh, these are the steps which should be used to compile and install ROOT-Sim in your environment. Please note that the installation procedure will not install ROOT-Sim system-wide, to simplify the removal process at the end of the reproducibility process. This might affect the path requirements to find the ROOT-Sim compiler (rootsim-cc) which is necessary to compile simulation models. In particular, configure will be instructed to install all the libraries and executables in the ~/rootsim/ folder of the user running the program.

To compile, these are the following steps:

cd runtime
./autogen.sh
./configure --enable-mpi --prefix=$HOME/rootsim/
make
make install

To make the path to rootsim-cc available, you can run:

export PATH=$PATH:/home/YOUR-USERNAME/rootsim/bin

where YOUR-USERNAME should be replaced with your username. The scripts used to re-run the experiments will try to discover the correct path on their own, if compile-and-install.sh is used.

The general steps to compile a simulation model are:

cd runtime/models/
cd model-folder
makeThis will generate an executable

where model-folder should point to one of the models in the repository. We have provided the compile-models.sh script which will automatize the process for all models used in the experimental assessment in the paper.

Running ROOT-Sim Models

To run a simulation model on a single machine, this is the command to be used:

./model --wt <n-threads> --lp <n-lps>

where <n-threads> should be replaced with the number of threads to be used for the simulation, and <n-lps> with the number of LPs which should be managed by the ROOT-Sim runtime environment. There is a special option for the --wt flags: if specifying --wt auto, the runtime environment will discover the number of available CPU cores on the machine, and use a corresponding number of threads. For the sake of performance, ROOT-Sim does not allow to use more threads than the available number of cores on the machine. This is something which should be taken into account when reproducing the results.

To run sequentially a model, ROOT-Sim implements a very optimized sequential algorithm implemented over a Calendar Queue. This optimized algorithm allows to stress test the parallel implementation. Indeed, running using the parallel algorithm with only one thread is not a safe way to assess the speedup of a parallel simulation engine, as there is the burden to support optimistic PDES anyhow. This cost in unnecessary, and might find

In order to go distributed, a common mpiexec command should be used. A possible syntax is:

mpiexec -n 2 --hostfile hosts --map-by node ./model --wt 2 --lp 16

This command will use two machines, (-n 2), described in the hostfile file. --map-by node is fundamental to use both machines, as on each machine the ROOT-Sim environment will spawn a suitable number of thread. This command uses two threads on each machine (a total of 4 threads), overall managing 16 LPs (8 LPs on each machine, 4 LPs per thread at the beginning of the simulation, then ROOT-Sim will automatically rebalance the load at runtime).

Assisted Scripts for Reproducibility

We have provided a script run-tests.sh. This script will try to re-generate all the results. Please note that this is a stochastic simulation, in which the seed used for the RNG can significantly affect the results. In the paper, we have presented results averaged over multiple runs. Collecting all data took weeks of CPU power, so in these scripts we are just making a single run for each point---the overall execution time of the experiments can be anyhow long.

To configure the execution of experiments in a way which is suitable for your machine, we have provided a configure.json file in the archive. There, you can describe the set of runs to be carried out, depending on your hardware configuration. While the outcome can be different from the results presented in the original paper, it will allow the reviewer to appreciate the trends of the curves.

The configure.json file is organized as follows (you can fine tune according to your preferences):

{
  "single": [4, 8, 12, 16, 24, 28, 32],
  "multi": [3, 6, 12, 24, 48, 96],
  "nodes": ["192.168.45.54", "192.168.45.50", "192.168.45.62"],
  "mpi_implementation": "mpich"
}

In the paper, we have run experiments both on a single machine and in a distributed environment. The single list will define how many runs with different core counts will be carried out on a single machine. The single machine used is the one on which the run-tests.sh script is launched. Each element in the single list tells how many cores should be used.

The multi list tells how many cores should be used in the parallel run. The nodes list tells the IP addresses of the machines to be used for the experiment. We expect that these machines can be reached over ssh with no password (a suitable ssh key should be installed) from the machine on which run-tests.sh is launched---this is the baseline configuration of MPI runtime environments, anyhow. The first IP in the nodes list must be the IP of the machine on which the run-tests.sh script is launched.

The mpi_implementation variable can be set to either mpich or openmpi: this will affects the commands generated by the script to run distributed simulations using the two different libraries.

We again stress that running all experiments takes time! The run-tests.sh is written so that the overall execution can be killed at any time. The runs already completed are cached in the current folder, so re-launching run-tests.sh should resume from the point at which the script was interrupted. The data folder created by the script caches these results. The availability of a subfolder in data tells the script that one specific run was completed, to if the data folder is removed (or a subfolder) the script will re-run the experiments.

Only a note: the run-tests.sh script will eventually have to specify the --wt flag to the ROOT-Sim runtime environment. To this end, the script attempts to divide evenly the number of threads specified in the multi list by the number of machines specified in the nodes list. We ask the reviewer to configure the multi and nodes lists in a way that each point in the multi list is a multiple of the nodes used. Overall, each machine must have a compliant number of cores. This is an oversimplification over the deploy model of ROOT-Sim, but this allows us to simplify the reproducibility process.

At the end of the execution of the script, the replicated images and tables will be located in the data subfolder.

This has been the best organization of the reproducibility of the results which we have been able to achieve in the limited amount of time for the initiative. If you encounter any problem, we are perfectly available to respond and help you at fixing the possible issues. Thanks again for the time spent of the reproducibility initiative!