ReFAIR: Toward a Context-Aware Recommender for Fairness Requirements Engineering

ReFair is a novel solution designed to assist Data Scientists and Software Engineers in building fair Machine Learning (ML) solutions. Its primary objective is to recommend potential sensitive features that could affect the fair behavior of an ML-intensive system during development.

The main idea behind ReFair is to analyze a User Story by identifying the application domain and the potential ML tasks required. Based on this information, ReFair suggests possible sensitive features for each task detected that are typically impactful in the identified domain.

As you can see, our approach is composed of two different machine learning modules:

• Single Label Classifier: This module takes a User Story as input and has the main objective of detecting the Application Domain of the story.
• Multi Label Classifier: This module takes a User Story as input and has the main objective of detecting the Possible Learning Tasks related to it.

Finally, once our approach detects the tasks and domain, it consults a specific mapping that we have built based on literature [1]. The mapping suggests possible Sensitive Features that the tool should pay attention to.

To develop an initial prototype of our approach, we created a Synthetic Dataset consisting of Machine Learning-specific User Stories. We then trained a supervised XGB CLASSIFIER for domain detection and a supervised Linear SVC Label Powerset multi-label classifier for task detection using this dataset. Finally, we developed an integrated console script that takes a User Story as input and displays the sensitive features for each task-domain pair.

For further specific information, we kindly invite you to read our research paper, which can be found at this link.

Repository Structure

Our Repository is structured into four subdirectories:

1. Starting Assets: This contains the resources we used to develop our approach [1, 2], which we discussed in our paper; the main contents of this subfolders are:
1. The original datasets ontology provided by Fabris et al.;
2. The ML techniques dictionary used to generate the user stories provided by Duran-Silva et al.;
2. Synthetic User Stories: This contains the synthetic dataset we used to train our classifiers, as well as the basic dictionaries we used to create it. The main contents of this subfolders are:
   1. The user stories dataset;
   2. The Domains and ML Tasks dictionary we used to create the user stories;
   3. The dataset validation script and the validation resources we used to validate it.
3. Source Code: This contains the source code related to the selection of the classification models and the integrated ReFair bash script; The main contents of this subfolders are:
   1. The domain classification analysis script and resources used to train the ML Model for the domain classification models.;
   2. The machine learning classification analysis script and resources used to train the ML Model for the machine learning tasks classification models.;
   3. The feature extraction script and the knowledge mapping regarding the relations between the application domains, the ML tasks and the sensitive features;
   4. The integrate ReFAIR executable script.
4. Experimental Results: This contains experimental reports regarding models selection and approach validation. The main contents of this subfolders are:
1. The results of the domains classification analysis;
2. Theresults of the tasks classification analysis;
3. The data, scripts and results related to the overall ReFair pipeline evaluation, in terms of input User Stories, and sensitive features suggested by ReFair
4. The data, scripts and results regarding the additional experiment we performed with the practitioners.

Please considered that in the appendix subfolders, are reported specific readme files to help the reader in contents visualization and understanding. In particular, to replicate the study, run the analysis, and obtained the same results, please make sure to include in the same directories of the main scripts the required resourses in terms of: python dependencies, datasets, and CSVs. To ensure a good level of simplicity, all analysis scripts have been coded via python notebooks, the execution of which simply requires having installed a compatible tool such as Visual Studio Code and a version of Python no lower than 3.9. Alternatively, it is possible to import the scripts and resources into Google Colab.