Software Open Access
Torralba, Alvaro; Seipp, Jendrik; Sievers, Silvan
This bundle contains all scripts required to reproduce the IPC Autoscale
benchmarks as described in the ICAPS 2021 paper "Automatic Instance
Generation for Classical Planning" by Álvaro Torralba, Jendrik Seipp and
The latest version of the Autoscale tool can be found at: https://github.com/AI-Planning/autoscale
The latest version of the Autoscale benchmarks can be found at https://github.com/AI-Planning/autoscale-benchmarks
* Directory "optimization" contains the main bulk of the experiment
scripts for all phases of the instance generation approach described in
the paper. See usage instructions below.
* Directory "pddl-generators" contains all generators of IPC domains that
were used for this paper (and some additional ones).
* Directory "planners" contains source code as well as Singularity recipe
files for all planners used in the paper. It also contains two scripts
for building all the corresponding Singularity images.
* Directory "logfiles" contains the logs of the `benchmark-optimization.py`
script for the results of the paper.
* Directory "report" contains some additional scripts and Jupyter
notebooks used to manually inspect the results of preliminary
experiments. They can show some interesting additional data, but they
are not intended to be maintained.
# Tested versions
- Python 3.7
- Singularity 3.5
- SMAC 0.12.3
- CPLEX 12.10
# Installation instructions
We recommend running Autoscale in a [Conda environment](https://conda.io).
The following commands create a Conda environment with the necessary
libraries and install the required Python packages (including SMAC).
conda create --name autoscale python=3.7 gxx_linux-64 gcc_linux-64 swig
conda activate autoscale
pip install -r requirements.txt
On Debian-based systems, probably the easiest way to install Singularity
is to use the [Singularity Debian
package](https://packages.debian.org/singularity-container). If that
doesn't work, you might have to [compile Singularity from
# Build planner images:
Install CPLEX 12.10 and add the python/3.7/ subfolder to the PYTHONPATH. By default this
is installed in "/opt/ibm/ILOG/CPLEX_Studio1210", so one has to set PYTHONPATH to
Before running the optimization, the following steps are needed:
* Edit `domain_configuration.py` to add the description of any new domain that you want to configure.
* Edit `planner_selection.py` to add the state-of-the-art planners that are relevant for
each domain. The script `select-best-planner-per-domain.py` can help to automatically
select a set of planners per domain from the results of earlier experiments.
Then, the following two steps must be performed:
1. Run `linear.py` to perform the SMAC optimization and generate sequences.
The `*smac*.py` scripts show how to parallelize this step on a computer cluster with Lab.
Note that the --database option allows to provide the JSON file from a previous SMAC
run to re-use all precomputed times for previously tested parameter configurations.
(see 2020-12-08-A-smac-opt-2014-20h.py for example)
2. Gather all results from one or multiple SMAC runs into a single properties JSON file.
This is done by running the last step of the `*smac*.py` scripts.
3. Run benchmark-optimization.py to generate a benchmark set. Use the `--database` option to provide
the properties file obtained from step 2.
`generate-benchmarks-2020-12-10.sh` contains an example on how to run the script.
# Experiment data
The optimization/results folder contains JSON files with the results of the experiments.
The results table in the paper is generated via the scripts `report-gather-dataset.py` and
|All versions||This version|
|Data volume||2.8 GB||2.8 GB|