Large-Scale Benchmarking of Metaphor-Based Optimization Heuristics -- Reproducibility Files

This repository contains the files needed to reproduce the data, analysis and visualizations from the paper 'Large-Scale Benchmarking of Metaphor-Based Optimization Heuristics'.

Data collection

For the data collection, we make use of the IOHexperimenter library to access the BBOB functions. In addion, we rely on the set of libraries discussed in the paper. For the full set of libraries used during the benchmarking experiment, please refer to reqs_benchmark.txt (in the Benchmarking folder).

The scripts used throughout the data collection are located in the 'benchmarking' directory. We have 1 file for each set of algorithms (expect mealpy and NIAPY, which are combined in the 'bestiary' file). Running these scripts generates a large amount of IOHprofiler-compatible files, which are availalbe in the 'Raw.zip' (Note: some of the libraries have their raw data split between multiple folders, if you want to run the full analysis, please merge these folders into one before processing). When running these scripts, please make sure to set the relevant FOLDER-variables at the top of the script!

Data processing

The raw data can be processed both for anytime performance (AOCC) and fixed-budget performance using the respective 'processing_METHOD.py' scripts. This generates several large csv-files, which are available in 'processed_data.zip'

Data analyis and visualization

The full set of analysis and corresponding visualizations can be reproduced using the 'Vizualization.ipynb' notebook. This notebook can use the processed performance data from the previous step, or the next level of processing with resulted in the remaining csv-files in this repository (this step is explained in the notebook itself).