Replication package for our article submitted to Transactions on Software Engineering and Methodology (TOSEM) Journal: "Retrospective Cohort Study In Action: A Pilot Investigation on the SonarQube's Impact on Issue Resolution Time"

This replication package contains the Python, R, and Jupyter Notebook code, figures, and datasets used for the data collection, preprocessing, matching, and statistical analysis of this study.

The study compares development velocity in Apache projects with and without SonarQube. Its follow-up period runs from 1 January 2021 up to, but not including, 1 January 2023.

NOTE: The hyperlinks do not work in Zenodo, they only work if the repository is downloaded and the README file is opened locally.

Getting started

These instructions will help you run the replication package locally.

Prerequisites

The statistical-analysis workflow requires R 4.5.1. Reconstructing the dataset additionally requires Python 3.10.20, Git, JupyterLab, scc, and credentials for the GitHub and SonarCloud APIs.

Follow INSTALL.md to install the required software, create isolated Python and R environments, configure the data paths and credentials, and verify the setup.

All commands and notebooks must be run from the repository root.

Structure of the replication package

The replication package provides source code, notebooks, study data, and generated figures.

Codes

The folder codes/ contains the Python scripts used for repository collection and preprocessing:

00_commons.py
06_clone_projects.py
07_get_commits.py
08_get_developer_experience.py
09_collect_contributors.py
10_collect_repo_experience.py
11_get_repo_confounders.py
13_calculate_velocity.py
15_format_data_for_analysis.py
16_merge_attributes.py
17_aggregate_experience.py

The folder codes/R_codes/ contains the R environment helpers and statistical-analysis scripts:

00_install_R_environment.R
00_analysis_setup.R
18_descriptive_analysis.R
19_crude_analysis.R
20_data_transformation.R
21_confounder_matching.R
22_regression_analysis.R

Notebooks

The folder notebooks/ contains the interactive data-collection and preprocessing stages:

01_collect_apache_projects.ipynb
02_collect_sonarqube.ipynb
03_collect_github_issues.ipynb
04_collect_jira_issues.ipynb
05_collect_github_commits.ipynb
12_prepare_project_data.ipynb
14_build_base_dataset.ipynb

Data

The folder datasets/ contains the archived inputs, intermediate files, and final analysis dataset:

JiraDataCrawler/
commonIssueFolder/
data_analysis_R/
experience/
githubIssueTables(TimeStamp)/
resultFiles/
scc_results/
sonarQubeData/
testCommits/
repositories_list.txt

The main analysis-ready dataset is:

datasets/data_analysis_R/SQ_cohort_data_in_days.csv

It contains 336 analyzed repositories and the exposure, outcome, and confounder variables consumed by the R scripts.

Figures

The folder Article_figures/ contains the manuscript figures and supporting plots:

CohortMethodology/
Matching_plots/
Results/
RetrospectiveCohort/
StudyDesign/
mikelFigures/
regression_plots/

Environment and license files

The repository root contains:

.env.example
.python_version
.R_version
requirements.txt
R_PACKAGES.txt
DATA_LICENSE
SCRIPT_LICENSE
INSTALL.md
README.md

requirements.txt records the Python dependencies, while R_PACKAGES.txt records the direct R dependencies. Consult SCRIPT_LICENSE for the source-code license and DATA_LICENSE for the dataset license.

Running the code

Two replication routes are available:

• Analysis replication: use the archived final dataset and run stages 18–22. This is the recommended route for reproducing the reported statistical analysis.
• Full reconstruction: run stages 1–22 to recollect and preprocess the data before performing the analysis.

Live GitHub, Jira, and SonarCloud data can change after the study period. Therefore, full reconstruction using live services may not reproduce the archived snapshot exactly. Use the archived Zenodo data for historical replication.

Stage 1: Collect Apache projects

• Run notebooks/01_collect_apache_projects.ipynb.
• Queries the GitHub API and selects the Apache repositories considered by the study.
• Generates datasets/resultFiles/cleanRepos070223.csv.

Stage 2: Collect SonarQube data

• Run notebooks/02_collect_sonarqube.ipynb.
• Queries SonarCloud to identify projects with SonarQube data.
• Generates the project and issue data under datasets/sonarQubeData/.

Stage 3: Collect GitHub issues

• Run notebooks/03_collect_github_issues.ipynb.
• Collects issue histories from GitHub.
• Stores one or more project tables under datasets/githubIssueTables(TimeStamp)/.

Stage 4: Collect Jira issues

• Run notebooks/04_collect_jira_issues.ipynb.
• Uses datasets/JiraDataCrawler/jira_data_sources.json to query Jira.
• Stores the issue tables under datasets/JiraDataCrawler/1.DataDownload/.

Stage 5: Collect GitHub commits

• Run notebooks/05_collect_github_commits.ipynb.
• Collects commit information for the projects selected in stages 1 and 2.
• Stores the study-period results under datasets/testCommits/.

Stage 6: Clone the selected repositories

• Run:

  python codes/06_clone_projects.py
  

• Clones the repositories listed in datasets/repositories_list.txt.

• The clone location defaults to repositories/ and can be changed with REPLICATION_REPOSITORY_DIR.

Stage 7: Extract repository commit histories

• Run:

  python codes/07_get_commits.py
  

• Extracts commit histories from the cloned repositories.

• Stores the results under datasets/repoExperience/.

Stage 8: Calculate developer experience

• Run:

  python codes/08_get_developer_experience.py
  

• Measures developer activity before and during the follow-up period.

• Stores the results under datasets/experience/.

Stage 9: Collect contributor information

• Run:

  python codes/09_collect_contributors.py
  

• Queries the GitHub API for contributor metadata.

• Generates datasets/contributors_data.json.

Stage 10: Collect repository experience

• Run:

  python codes/10_collect_repo_experience.py
  

• Queries GitHub and calculates repository-level experience measures.

• Stores the results under datasets/repoExperience/.

Stage 11: Collect repository confounders

• Run:

  python codes/11_get_repo_confounders.py
  

• Uses scc on the cloned repositories to measure source-code size and complexity.

• Stores the measurements under datasets/scc_results/.

Stage 12: Prepare the project data

• Run notebooks/12_prepare_project_data.ipynb.
• Merges and cleans the GitHub and Jira issue tables.
• Generates the unified project issue files and datasets/resultFiles/projects_before_velocity.csv.

Stage 13: Calculate development velocity

• Run:

  python codes/13_calculate_velocity.py
  

• Calculates project development velocity from datasets/commonIssueFolder/.

• Generates datasets/resultFiles/velocity_data(githubIssues)30032023.csv.

Stage 14: Build the base dataset

• Run notebooks/14_build_base_dataset.ipynb.
• Reloads the stage-12 intermediate data and combines it with the stage-13 velocity results.
• Generates datasets/resultFiles/projectFinalFile.csv.

Stages 12 and 14 are separate so the pipeline does not depend on retaining an in-memory notebook object while stage 13 runs.

Stage 15: Format the analysis dataset

• Run:

  python codes/15_format_data_for_analysis.py
  

• Selects Java projects, formats the variables, and creates datasets/data_analysis_R/SQ_cohort_data_in_days.csv.

Stage 16: Merge repository attributes

• Run:

  python codes/16_merge_attributes.py
  

• Adds size and complexity measurements to the analysis CSV.

• Excludes projects for which these repository confounders are unavailable.

Stage 17: Aggregate developer experience

• Run:

  python codes/17_aggregate_experience.py
  

• Adds the normalized developer-experience measures to the analysis CSV.

Stage 18: Descriptive analysis

• Run:

  Rscript codes/R_codes/18_descriptive_analysis.R
  

• Produces descriptive statistics and exploratory diagnostics.

Stage 19: Crude analysis

• Run:

  Rscript codes/R_codes/19_crude_analysis.R
  

• Performs the unadjusted comparison between the SonarQube and non-SonarQube groups.

Stage 20: Data transformation

• Run:

  Rscript codes/R_codes/20_data_transformation.R
  

• Evaluates distributional assumptions and transformations used by the later analysis.

Stage 21: Confounder matching

• Run:

  Rscript codes/R_codes/21_confounder_matching.R
  

• Evaluates confounder balance and performs coarsened exact and nearest-neighbor matching.

• Writes generated result plots under Article_figures/Results/.

Stage 22: Adjusted regression analysis

• Run:

  Rscript codes/R_codes/22_regression_analysis.R
  

• Performs multicollinearity checks, matching diagnostics, principal-component analysis, and adjusted regression modelling.

• Prints statistical tables and model summaries to the console and generates supporting plots.

Configuration notes

The scripts read configuration from environment variables. They do not load .env automatically.

• REPLICATION_DATA_DIR: dataset root; defaults to datasets/.
• REPLICATION_REPOSITORY_DIR: repository clone directory; defaults to repositories/.
• REPLICATION_REPOSITORIES_LIST: repository URL list; defaults to datasets/repositories_list.txt.
• GITHUB_TOKEN: required by GitHub collection stages.
• SONAR_TOKEN: required by the SonarCloud collection stage.
• REPLICATION_NOTIFICATION_SENDER, REPLICATION_NOTIFICATION_RECIPIENT, and REPLICATION_NOTIFICATION_PASSWORD: optional completion-email settings.

Copy .env.example to .env if a local reference file is useful, but export the required values in the shell that launches Python or Jupyter. Never commit credentials.

The follow-up interval is defined in codes/00_commons.py as [2021-01-01, 2023-01-01) in UTC. The R workflow uses set.seed(2026) through codes/R_codes/00_analysis_setup.R.