Code and Benchmarks from the paper "Automatic Instance Generation for Classical Planning"'

This bundle contains all scripts required to reproduce the IPC Autoscale
benchmarks as described in the ICAPS 2021 paper "Automatic Instance
Generation for Classical Planning" by Álvaro Torralba, Jendrik Seipp and
Silvan Sievers.

The latest version of the Autoscale tool can be found at: https://github.com/AI-Planning/autoscale

The latest version of the Autoscale benchmarks can be found at https://github.com/AI-Planning/autoscale-benchmarks

* Directory "optimization" contains the main bulk of the experiment
  scripts for all phases of the instance generation approach described in
  the paper. See usage instructions below.

* Directory "pddl-generators" contains all generators of IPC domains that
  were used for this paper (and some additional ones).

* Directory "planners" contains source code as well as Singularity recipe
  files for all planners used in the paper. It also contains two scripts
  for building all the corresponding Singularity images.

* Directory "logfiles" contains the logs of the `benchmark-optimization.py`
  script for the results of the paper.

* Directory "report" contains some additional scripts and Jupyter
  notebooks used to manually inspect the results of preliminary
  experiments. They can show some interesting additional data, but they
  are not intended to be maintained.

# Tested versions

 - Python 3.7
 - Singularity 3.5
 - SMAC 0.12.3
 - CPLEX 12.10

# Installation instructions

We recommend running Autoscale in a [Conda environment](https://conda.io).
The following commands create a Conda environment with the necessary
libraries and install the required Python packages (including SMAC).

    conda create --name autoscale python=3.7 gxx_linux-64 gcc_linux-64 swig
    conda activate autoscale
    cd optimization
    pip install -r requirements.txt

## Singularity

On Debian-based systems, probably the easiest way to install Singularity
is to use the [Singularity Debian
package](https://packages.debian.org/singularity-container). If that
doesn't work, you might have to [compile Singularity from
source](https://sylabs.io/guides/3.5/user-guide/).

# Build planner images:

    export DOWNWARD_BENCHMARKS=<path/to/benchmarks/repo>
    cd planners
    ./build-all-images.sh

## CPLEX

Install CPLEX 12.10 and add the python/3.7/ subfolder to the PYTHONPATH. By default this
is installed in "/opt/ibm/ILOG/CPLEX_Studio1210", so one has to set PYTHONPATH to
":/opt/ibm/ILOG/CPLEX_Studio1210/cplex/python/3.7/x86-64_linux/"

# Usage

Before running the optimization, the following steps are needed:
* Edit `domain_configuration.py` to add the description of any new domain that you want to configure.

* Edit `planner_selection.py` to add the state-of-the-art planners that are relevant for
  each domain. The script `select-best-planner-per-domain.py` can help to automatically
  select a set of planners per domain from the results of earlier experiments.

Then, the following two steps must be performed:
1. Run `linear.py` to perform the SMAC optimization and generate sequences.
   The `*smac*.py` scripts show how to parallelize this step on a computer cluster with Lab.

   Note that the --database option allows to provide the JSON file from a previous SMAC
   run to re-use all precomputed times for previously tested parameter configurations.
   (see 2020-12-08-A-smac-opt-2014-20h.py for example)

2. Gather all results from one or multiple SMAC runs into a single properties JSON file.
   This is done by running the last step of the `*smac*.py` scripts.

3. Run benchmark-optimization.py to generate a benchmark set. Use the `--database` option to provide
   the properties file obtained from step 2.
   `generate-benchmarks-2020-12-10.sh` contains an example on how to run the script.

# Experiment data

The optimization/results folder contains JSON files with the results of the experiments.
The results table in the paper is generated via the scripts `report-gather-dataset.py` and
`report-create-table.py`.