Differentiating Refactoring Practices: A Comparative Analysis of ML and Non-ML Frameworks

Research  Overview:
This research is a longitudinal empirical study of refactoring practices between machine learning (ML) frameworks and non-ML frameworks in the field of software engineering. We mined over 1-million commits and over 4-million refactorings from 130 ML frameworks and 800 non-ML frameworks (i.e. 65 ML and 400 non-ML Java frameworks; 65-ML and 400 non-ML Python frameworks). We used the tool RefactoringMiner to extract Java and Python refactorings respectively, while we implement a custom Python script using GitHub API to extract the commits from their repositories. We further divided the commit and refactoring extraction into stage grouping (early, middle and late) using the emperical proposition of prior study.

Notably, this research is motivated to understand how refactoring differs per development stage in ML vs non-ML framework, and how these findings can benefit software engineering practitioners in the area of software quality assurance. This research is primarily quantitative as we only analyzed the refactorings that were extracted by RefactoringMiner.

Follow the guide below to reproduce our study findings: 

(A) Requirements

• Build RefactoringMiner version 3.1.2 from Github to extract Python and Java refactorings
• Build Lizard framework version 1.21.3 to evaluate code quality before and after refactoring   
• Install all libraries in the requirements.txt

(B) Dataset Availability 

• The zip file labeled: "Datasets_and_analysis" contains the extracted refactoring and commit JSON files for both Java and Python  frameworks.
• All commits JSON can be located in the folder path ./project_commits", while refactoring .JSON can be located in folder "./raw_json_data".

(C) Implementation Steps

• First,  generate a GitHub access token to have access to call the GitHub API when using RefactoringMiner, and our custom Python scripts. Then extract the data inside "Datasets_and_analysis" zip file  

Step 1: To run RefactoringMiner, you can build as a Gradle or Maven project. Although, we provide our custom Python script (refactoringminer.py) that integrates the RefactorinMiner's data extraction function in order to extract refactorings of multiple frameworks in a loop, you can as well visit their official Github page for more implementation details.

• run refactoringminer.py to extract refactorings of studied subjects
• run extract_commit.py to extract commits from Github of studied subjects

Step 2: run python file "./pull_data_raw_json.py" to extract the required nodes from the commits and refactoring JSON into CSV files inside the folder named "extracted_refactorings". 

Step 3: run "group_into_stages.py" to group the refactorings and commits into their respective development stages (early, middle and late).

Step 4: Analysis: We provide the implementation script for each RQ as presented below:  

• Python scripts with prefix "rq1_", "rq2_", and "rq3_" represents implemetation scripts to analyze and generate figures for for RQ1, RQ2, and RQ3 respectively .
• Other Python script without the prefix is labeled to identify their functionality 
• lizard_refactoring_impact.py contains all the Python implementation script for Lizard framework which we used to estimate code quality of the subject systems before and after refactotoring

Step 5: Manual investigation records for RQ1, RQ2, and RQ3 can be found in the folder labeled "manual_verification" folder.

Note:

• All Figures generated by the implementation scripts can be found inside the generated "./outputs" folder
• Table data are generated based on analysis for each RQ