# SIMBA: Status of IMBAlance, a robust and generalizable measure for data imbalance

This repository contains the code used for the paper  ***"Data Imbalance Measures, a review and a way forward"***.

## Folder structure

- <ins>*data/*</ins>

This folder contains the processed 70 datasets used for the tests on real data. Each dataset is a folder containing two files: *df_allfeatures.csv* for the features and *labels.csv* for the associated labels.

- <ins>*table/*</ins>

This folder contains descriptive tables of the 70 datasets used (in .csv format). The subfolder results/ contains all the results files (for the correlation analysis and ablation study) to plot/display the results later on.

- <ins>*current folder*</ins>

The current folder contains all code files (in python), the present README file, and a requirements text file to give information on the libraries used to run this code. 

## Code

- *<ins>data_processing.py</ins>*:

This file contains all the functions that were used to get the files present in the data/ folder.

- *<ins>imbalance_measures.py</ins>*:

This file contains all the functions necessary to calculate the seven imbalance measures, namely Imbalance Ratio (IR), Adjusted Imbalance Ratio (Adj-IR), Entropy of Class Proportions (C1), Imbalance Degree (ID), Likelihood Ratio Imbalance Degree (LRID), Imbalance Factor (IF), and the Status of IMBAlance (SIMBA). This file is necessary to run most functions from main.py.

- *<ins>main.py</ins>*:

This file contains all the functions necessary to perform classification tasks on both synthetic and real datasets, calculate and plot the correlation between the seven imbalance measures and the obtained macro average f1-scores/g-means. Following is a description of how to use the main functions that were used for the paper.

## Usage

Depending on the part of the code you would like to run, you can set the variables "action" (to select which function to run), "eval" (choose between "f1" and "gmean") and "N" (number of runs for the synthetic datasets). They are declared after the "if \_\_name\_\_ == "\_\_main\_\_".  

- *make_DS_descriptive_table(all_ds, col_label, only_class_freq)*: 
    
    This function makes a descriptive table of the datasets contained in the data/ folder and saves it as a .csv in the table/ folder.
    
    To run it, set action = "table_ds".
        
        @all_ds: list of Strings, list of the name of all datasets that should be in the table. 

        @col_label: (optional) String, default='label', name of the labels column in the datasets. 
        [WARNING]: the name of the labels column should be the same in all the datasets!

        @only_class_freq: (optional) boolean, default=False, 
        if False, the table contains: dataset name, number of classes, number of features, total number of points, mean and sd of class frequencies, results of all 6 imbalance metrics (IR, adjIR, ID, LRID, IF, SIMBA).
        if True, the table contains:  dataset name, number of classes, number of minority classes, list of class frequencies.

- *run_synthdata(N, data_distrib, feature_distrib, tot_ft, i_range, title, corr, varying, eval_method)*: 
    
    This function creates and runs the tests on synthetic datasets. Either computes metrics and f1-scores/gmeans for each dataset, and plots trend of the metric, PCC and SRCC between metrics scores and f1-scores/gmeans (corr=True). Or, computes only the metrics and return them, but no plot is done (corr=False). The dataset generated can vary on data
    distribution or on the feature distribution (informative, redundant, not informative).

    To run it with varying data, set action = "synth_data". To run it with varying features, set action = "synth_ft".
        
        @N: integer, number of random runs (set to 100 by default, but it can be changed in the code). For each run, a different random state is picked.

        @data_distrib: list of integers size 3, we use datasets with 3 classes. data_distrib gives the number of data points in each class. If one element of the list == 0, this element is the varying one (it will take all values in i_range). 

        @feature_distrib: list of integers size 3, this represents [#informative ft, #redundant ft, #not informative ft], if varying is not on data but on ft, one of these might be equal to "-1" and will therefore be the varying element. tot_ft can also vary according to this.

        @tot_ft: integer, total number of features. It is equal to sum(feature_distib) when data distribution is varying. If ft is varying, then it is possible that it is not fixed (i.e. == -1), then it will vary with i. 

        @i_range: list of integers, size 3, i is the varying factor, whether for data or feature distribution. i will take any values in the specified range [min (incl.), max (excl.), step].

        @title: String, title of the plots, only used if corr==True.

        @corr: boolean, if True, metrics and f1-scores are computed for each dataset for each i, and 3 plots are made: trend of the metrics when i is varying, PCC and SRCC between metrics scores and f1-scores. 
        If False, the f1-scores are not computed, only the metrics scores, which are returned. 

        @varying: String, defines what is varying between "data" or features. 
        If == "data", data distribution is varying.
        If != "data", it is considered that the features are varying.

        @eval_method: String, defines the classification evaluation metric: either == "f1" for f1-score or != "f1" for gmean (set to "f1" by default, but it can be changed in the code).

- *run_synthdata_plot(N, varying, eval_method, from_file)*: 
    
    If you have not run the run_synthdata function, set from_file to False, and this function creates and runs all the tests on synthetic datasets, but only on the metrics (no correlations). Then, the function will generate plots for each imbalance metrics and show their trend on the three data distribution variations (if varying=="data") or the three features distribution variations (if varying!="data"). If from_file == True, the plots will be made from the files saved with the run_synthdata function.

    To run it with varying data, set action = "synth_data_plot". To run it with varying features, set action = "synth_ft_plot".
    
        @N: integer, number of random runs (set to 100 by default, but it can be changed in the code). For each run, a different random state is picked.
        
        @varying: String, defines what is varying between "data" or features. 
        If == "data", data distribution is varying.
        If != "data", it is considered that the features are varying.

        @eval_method: String, defines the classification evaluation metric: either == "f1" for f1-score or != "f1" for gmean (set to "f1" by default, but it can be changed in the code).

        @from_file: Boolean, default=False. When False, the synthetic data analysis is run with run_synthdata(). When True,
                the run_synthdata() function has already been run and the plots are generated from the files saved with
                this function.

- *run_realdata(classif, eval_method)*: 

    This function runs the tests on all real datasets (contained in data/ folder). Metrics and f1-scores/gmeans are computed for each dataset.

    To run it, set action = "real_data".
    To obtain the regression plots, run the "real_data_plot" action after running the "real_data" one. It will use the show_realdata_res() function and generate plots of the PCC between metric and f1-score/gmean for all datasets, only multiclass datasets, and only binary datasets are generated for each imbalance metric (IR, adjIR, C1, ID, LRID, IF, and SIMBA).

        @classif: (optional) String, default="SVM", chosen classifier, can take the following values: "SVM", "LDA", "kNN", "RF", and "MLP".
        [WARNING!] any other value will lead to an error and the program will be stopped.

        @eval_method: String, defines the classification evaluation metric: either == "f1" for f1-score or != "f1" for gmean (set to "f1" by default, but it can be changed in the code).

- *run_real_ablation(classif, eval_method)*: 

    This function runs the ablation study on SIMBA on all real datasets (contained in data/ folder). SIMBA scores and f1-scores/gmeans are computed for each SIMBA version: SIMBA whole, SIMBA without normalization, SIMBA without feature importance, and SIMBA without feature redundancy. 

    To run it, set action = "ablation". To obtain the regression plots, run the "ablation_plot" action after running the "ablation" one. It will use the plot_real_ablation() function and generate plots of the PCC between the 4 versions of SIMBA and f1-score/gmean for all datasets.

        @classif: (optional) String, default="SVM", chosen classifier, can take the following values: "SVM", "LDA", "kNN", "RF", and "MLP".
        [WARNING!] any other value will lead to an error and the program will be stopped.

        @eval_method: String, defines the classification evaluation metric: either == "f1" for f1-score or != "f1" for gmean (set to "f1" by default, but it can be changed in the code).

- *data_complexity_analysis()*: 

    This function runs the control study with multiple data complexity measures. It investigates the influence of data complexity on the different imbalance measures and classification performance on all real datasets. 

    To run it, set action = "complexity".
    

## Requirements

Python 3.11.9 was used for this project. For the version of the libraries used, please refer to the *requirements.txt* document. 

---

*Author: Julie R. Pivin-Bachler, julie@pivin-bachler.eu*