Switching between Numerical Black-box Optimization Algorithms with Warm-starting Policies - Reproduction artifacts

# Reproducibility instructions

This document details the steps to reproduce the results presented in the paper 'Chaining of Numerical Black-box Algorithms: Investigating the Impact of Warm-Starting and the Switching Point'

## Algorithm code
The code for the individual algorithms and the switching routines is included in the 'dynas.zip' folder.  

## Data collection
To collect the performance data for the static algorithms, the file 'data_collection.py' can be used. This file runs all 5 static algorithm on the function suite used in the paper, using the 'ioh' package. To run it, please modify the 'ioh_dir' variable to a place where the data should be stored. 
The resulting data is made available under the 'static.zip' file.

## Processing code
The code which processes the performance data into the list of usecases is part of the 'Processing_R' notebook. 
This results in a set of rds-files with the data in IOHanalyzer's internal format, and in a csv file ('Split_performances_relative.csv') with the selected usecases. 

This csv file can then be read in using the 'visualization' notebook, which can generate Figure 1 from the paper. This notebook can then be used to count the specific usecases, which is the data shown in Table 1. The csv-file containing the selected usecases ('usecases.csv') is also generated here.

The 'usecases' csv file can be used in the 'Data_collection_switch' notebook to run the dynamic configurations. This results in the 'Dynamic.zip' data, which can again be read using IOHanalyzer in the 'Processing_R' notebook. This then generates several csv-files, for both the static and dynamic case, containing the per-run hitting times ('RT_samples.zip'). These are then used in the 'Visualization' notebook to generate figure 3 and 4, as well as table 2. 

## Countour plots
The code for generating Figure 5 is included in the end of the 'Visualization' notebook.

## Performance comparison plot
Figure 6 has been generated using IOHanalyzer web-interface (iohanalyzer.liacs.nl), by uploading the files in 'cma_bfgs.zip'. 

## Switch point impact plot
Figure 7 has been generated using the 'split_experiment' notebook