IOH-Xplainer - code, experiments and results

README

Steps to reproduce the experiments / figures from the paper.

1. First we need to setup the environment. Unzip the file `iohxplainer.zip` in the root folder.
2. Now place the `pyproject.toml` file, `poetry.lock` file and the `README.md` file in the root directory.
3. Download the experiments and unpack them in the root directory.
4. Download the pkl files and place them in the experiments directory.
5. You should now have the following directory structure:
   1. /iohxplainer/
   2. /experiments/
      1. /experiments/cma_final_processed_new.pkl
      2. /experiments/de_final_processed_new.pkl
   3. /pyproject.toml
   4. /README.md
   5. /poetry.lock
6. Make sure you have Poetry 1.7.1 (or higher) and Python 3.10 installed.
7. Run `poetry install` in the root directory.
8. The experiments use modcma and modde, install them with poetry add modde modcma.
9. (OPTIONAL!!!)  Run all Modular DE or Modular CMA configurations using the (de|cma_es)_run-configurations.py file (in the experiments folder), this writes a pkl file as result. This step takes a month on a supercomputer with 120 cores. The processed results of this step are already available as the pkl files in this repository.
10. (OPTIONAL, ONLY RUN IF YOU RUN STEP 9) Pre-process the pickle files with poetry run python experiments/(de|cma_es)_process_pkl.py.
11. Analyse the performance data of all configurations using IOH-Xplainer poetry run python experiments/(de/cma_es)_analyze.py (Expected runtime=24 hours)
12. Compare the two frameworks using poetry run pythonexperiments/compare_de_cma.py. Writes the result as latex file (compare-new.tex). (Expected runtime=1 hour)
13. Perform automated algorithm configuration experiment using experiments/(de|cma_es)_AAC-notebook.ipynb files. (Expected runtime = 1 hour per file)