IOH-Xplainer - code, experiments and results

README

Steps to reproduce the experiments / figures from the paper.

1. First we need to setup the environment. Download and unzip the repository.zip.
2. Create a new directory called output (for storing all results).
3. You should now have the following directory structure:
   1. /iohxplainer/ (with .py files)
   2. /experiments/
      1. /experiments/cma_es_AAC-notebook.ipynb
      2. /experiments/de_AAC-notebook.ipynb
      3. /experiments/*.py files
   3. /output/
   4. /de_final_processed_new.pkl
   5. /cma_final_processed_new.pkl
   6. /ela-features-cma.pkl
   7. /ela-features-de.pkl
   8. /pyproject.toml
   9. /README.md
   10. /poetry.lock
4. Make sure you have Poetry 1.7.1 (or higher) and Python 3.10 installed.
5. Run `poetry install` in the root directory.
6. (OPTIONAL!!!)  Run all Modular DE or Modular CMA configurations using the (de|cma_es)_run-configurations.py file (in the experiments folder), this writes a pkl file as result. This step takes a month on a supercomputer with 120 cores. The processed results of this step are already available as the pkl files in this repository.
7. (OPTIONAL, ONLY RUN IF YOU RUN STEP 6) Pre-process the pickle files with poetry run python experiments/(de|cma_es)_process_pkl.py.
8. Analyse the performance data of all configurations using IOH-Xplainer poetry run python experiments/(de/cma_es)_analyze.py (Expected runtime=24 hours)
9. Compare the two frameworks using poetry run pythonexperiments/compare_de_cma.py. Writes the result as latex file (compare-new.tex). (Expected runtime=1 hour)
10. Perform automated algorithm configuration experiment using experiments/(de|cma_es)_AAC-notebook.ipynb files. (Expected runtime = 2 hours per file)