Introduction

This is the replication package for Fairabel, which includes the source code for Fairabel. The package contains three folders: 'ML,' 'DL,' and 'AnalysisResult', which contain the source code and experimental results of Fairabel in classical machine learning and deep learning scenarios.

'AnalysisResult' contains the anaysys results of Fairabel and baseline methdos, including the Faireal trade-off proportion when considering f1-score, each performance and fairness metric across different tasks and models, results of hypothesis test with multiple comparison protection.

'ML' and 'DL' folders contain Fairabel's source code. The 'results' folder in 'ML' and 'DL' folders contains the raw results raw results after applying Fairabel.

Each file in the 'results' folder is named according to the experiment setting, with the file extension indicating the file type. For example, a file named 'Fairabel_lr_adult_race.txt' refers to a logistic regression model that applied the Fairabel method to protect the race attribute in the Adult Census Income dataset.

Each file in the 'results' folder has 53 columns. The first column indicates the ML performance or fairness metric, followed by 50 columns of metric values from 50 runs, and the last two columns show the mean and standard deviation values of the 50 runs.

 

Experimental environment

 

We use Python 3.8 for our experiments. We use the IBM AI Fairness 360 (AIF360) toolkit to implement bias mitigation methods and compute fairness metrics.

 

Installation instructions for Python 3.8 and AIF360 can be found at https://github.com/Trusted-AI/AIF360. That page provides several ways to install it. We recommend creating a virtual environment for it (as shown below), because AIF360 requires specific versions of many Python packages which may conflict with other projects on your system. If you want to try other installation methods or encounter any errors during the installation process, please refer to the page (https://github.com/Trusted-AI/AIF360) for help.

 

Conda

 

Conda is recommended for all configurations. [Miniconda](https://conda.io/miniconda.html)

is sufficient (see [the difference between Anaconda and

Miniconda](https://conda.io/docs/user-guide/install/download.html#anaconda-or-miniconda)

if you are curious) if you do not already have conda installed.

 

Then, to create a new Python 3.8 environment, run:

 

```bash

conda create --name aif360 python=3.8

conda activate aif360

```

 

The shell should now look like `(aif360) $`. To deactivate the environment, run:

 

```bash

(aif360)$ conda deactivate

```

 

The prompt will return to `$ `.

 

Note: Older versions of conda may use `source activate aif360` and `source

deactivate` (`activate aif360` and `deactivate` on Windows).

 

Install with `pip`

 

To install the latest stable version from PyPI, run:

 

```bash

pip install 'aif360'

```

 

[comment]: <> (This toolkit can be installed as follows:)

 

[comment]: <> (```)

 

[comment]: <> (pip install aif360)

 

[comment]: <> (```)

 

[comment]: <> (More information on installing AIF360 can be found on https://github.com/Trusted-AI/AIF360.)

 

In addition, we require the following Python packages.

```

pip install sklearn

pip install numpy

pip install shapely

pip install matplotlib

pip install --upgrade protobuf==3.20.0

pip install fairlearn

```

 

Dataset

 

We use the five default datasets supported by the AIF360 toolkit. **When running the scripts that invoke these datasets, you will be prompted how to download these datasets and in which folders they need to be placed.** You can also refer to https://github.com/Trusted-AI/AIF360/tree/master/aif360/data for the raw data files.

 

Scripts and results

The repository contains the following folders:

* ```ML/``` contains the raw results and code of Fairabel using ML algorithms.

* ```DL/``` contains the raw results and code of Fairabel using DL algorithms.

* ```AnalysisResult/``` contains the statistical analysis results of Fairabel and baseline methods.

 

* ```results/``` in 'ML/' and 'DL/' folds contain the raw results of the models after applying Fairbel. Each file in these folders has 53 columns, with the first column indicating the metric, the next 50 columns the metric values of 50 runs, and the last two columns the mean and std values of the 50 runs.

 

Reproduction

You can reproduce the results from scratch. We provide a step-by-step guide on how to reproduce the results.

 

We obtain the ML performance and fairness metric values obtained by our approach, Fairabel (`ML/recor_50.py`). `recor_50.py` supports three arguments: `-d` configures the dataset; `-c` configures the ML algorithm; `-p` configures the protected attribute.

```

cd ML

python recor_50.py -d adult -c lr -p sex

python recor_50.py -d adult -c lr -p race

python recor_50.py -d compas -c lr -p sex

python recor_50.py -d compas -c lr -p race

python recor_50.py -d german -c lr -p sex

python recor_50.py -d german -c lr -p age

python recor_50.py -d bank -c lr -p age

python recor_50.py -d mep -c lr -p RACE

 

python recor_50.py -d adult -c rf -p sex

python recor_50.py -d adult -c rf -p race

python recor_50.py -d compas -c rf -p sex

python recor_50.py -d compas -c rf -p race

python recor_50.py -d german -c rf -p sex

python recor_50.py -d german -c rf -p age

python recor_50.py -d bank -c rf -p age

python recor_50.py -d mep -c rf -p RACE

 

 

python recor_50.py -d adult -c svm -p sex

python recor_50.py -d adult -c svm -p race

python recor_50.py -d compas -c svm -p sex

python recor_50.py -d compas -c svm -p race

python recor_50.py -d german -c svm -p sex

python recor_50.py -d german -c svm -p age

python recor_50.py -d bank -c svm -p age

python recor_50.py -d mep -c svm -p RACE

```