Evaluating Real-World Generalizability of Algorithm Selection Models

This Zenodo upload contains:

• Raw benchmark results (per algorithm, per dimension, per benchmark)

• Instance sample files used to compute ELA features

• Computed ELA feature tables

• Aggregated performance tables used in the algorithm-selection experiments

• Crossmatch test CSVs for distribution-comparison analyses

1. Top-level layout

At the top level you will find four benchmark folders, one folder with ELA features, several aggregated performance tables, and CSVs for crossmatch analyses:

BBOB/ Raw results + samples for BBOB benchmark
CEC/ Raw results + samples for CEC benchmark
ROB/ Raw results + samples for ROB benchmark
UAV/ Raw results + samples for UAV benchmark
ela/ Computed ELA feature tables (CSV)

performance_dim_6_noPRS.csv
performance_dim_6_no_agg_noPRS.csv
performance_dim_12_noPRS.csv
performance_dim_12_no_agg_noPRS.csv
performance_dim_18_noPRS.csv
performance_dim_18_no_agg_noPRS.csv
performance_dim_24_noPRS.csv
performance_dim_24_no_agg_noPRS.csv
performance_dim_30_noPRS.csv
performance_dim_30_no_agg_noPRS.csv

crossmatch_test.csv
crossmatch_test_cosine.csv
crossmatch_test_euclidean.csv

.ipynb_checkpoints/ (optional) notebook metadata folder

Top-level files explained

A) performance tables

• performance_dim_{d}_noPRS.csv
  Aggregated performance table for dimension d (mean aggregated over runs), excluding PRS.

• performance_dim_{d}_no_agg_noPRS.csv
  Non-aggregated performance table for dimension d (keeps per-run information), excluding PRS.

B) crossmatch tables

• crossmatch_test*.csv
  CSVs used for/produced by crossmatch-style distribution comparison tests (including cosine and euclidean variants).

2. Benchmark folders (ROB/, BBOB/, UAV/, CEC/)

Each benchmark folder follows the same structure and naming conventions.

Inside each of ROB/, BBOB/, UAV/, CEC/ you typically find:

• _readme Short benchmark-specific notes (if present)

• .ipynb_checkpoints/ (optional)

• res_<BENCH>_dim<d>_samples.csv

• res_<BENCH>_dim<d>budget20000<ALGO>.csv (multiple algorithms)

File types inside each benchmark folder

A) Instance samples (input for ELA)

• res_<BENCH>_dim<d>_samples.csv
  Instance sample data used to compute ELA features for benchmark <BENCH> in dimension <d>.
  These files are the inputs to the ELA computation step.

B) Raw algorithm results

• res_<BENCH>_dim<d>budget20000<ALGO>.csv
  Raw performance results for a specific algorithm <ALGO> on benchmark <BENCH>, dimension <d>,
  with evaluation budget 20000.

Placeholders:

• <BENCH> is one of: ROB, BBOB, UAV, CEC

• <d> is the problem dimension (e.g., 6, 12, 18, 24, 30)

• <ALGO> is the algorithm identifier (e.g., DE, PSO, LSHADE, etc.)

3. ELA features folder (ela/)

The ela/ directory contains computed ELA feature tables generated from the res_*_samples.csv files.

Typical naming convention:

• <BENCH>dim<d>scale<S>downsample<FLAG>.csv

Where:

• <S> indicates the scaling mode used during feature generation (0, 1, or y)

• <FLAG> indicates whether downsampling was applied (0/1)

4. AS_results_* folders

The AS_results folders contain the outputs of the algorithm selection experiments (produced by 2_AS.py). Files are saved per experimental setting, where the setting is encoded in the filename.

Each result file name follows the pattern:

TYPE_dim_<d>scale<S>downsample<True|False>.csv

Where:

• TYPE indicates what is stored in the file:

  • final_results: aggregated evaluation metrics and summary results for the AS experiment

  • predictions: per-instance predictions (e.g., selected algorithm / predicted best / scores depending on the run)

  • feature_importances: feature importance values from the trained model(s)

• d is the problem dimension (e.g., 6, 12, 18, 24, 30)

• S is the scaling mode used for the features/targets:

  • 0 = no scaling

  • 1 = min–max scaling of X and y

  • y = min–max scaling of y only

• downsample indicates whether downsampling was applied to the instance samples before computing ELA features:

  • True = downsampling applied

  • False = no downsampling

Typical files inside AS_results

• final_results_dim_<d>scale<S>downsample<...>.csv
  Summary metrics for the experiment setting (one row per model/strategy or per split, depending on configuration).

• predictions_dim_<d>scale<S>downsample<...>.csv
  Instance-level outputs for the experiment setting (e.g., chosen algorithm and/or predicted performance).

• feature_importances_dim_<d>scale<S>downsample<...>.csv
  Feature importance values computed from the trained model(s) for that setting.

5. Relationship between files (high level)

• Benchmark sample files:
  <BENCH>/res_<BENCH>_dim<d>_samples.csv
  are used to compute ELA feature tables in:
  ela/<BENCH>dim<d>scale<S>downsample<FLAG>.csv

• Benchmark raw algorithm result files:
  <BENCH>/res_<BENCH>dim<d>budget20000<ALGO>.csv
  are aggregated into the top-level performance tables:
  performance_dim{d}noPRS.csv and performance_dim{d}_no_agg_noPRS.csv

• crossmatch_test*.csv are used for distribution-comparison / crossmatch analyses across datasets or feature spaces