Published February 4, 2026 | Version v3
Dataset Open

Research Artifact – Cross-Project Flakiness: A Case Study of the OpenStack Ecosystem

Authors/Creators

  • 1. ROR icon Kyushu University

Description

This repository serves as the research artifact for the paper “Cross-Project Flakiness: A Case Study of the OpenStack Ecosystem”. It contains a dataset of flaky tests mined from OpenStack, along with their associated metadata, scripts for data collection, and the analysis logic to reproduce the findings in RQs 1–3.

Requirements & Warnings

  • Storage: This replication package is over 20 GB. Ensure you have sufficient disk space and a stable internet connection before downloading.
  • Hardware: Due to the size of the datasets, high-memory (RAM) is recommended for running the analysis notebook.
  • Database: A working installation of MongoDB and the mongorestore utility is required.

Replication Workflow

To replicate the results presented in the paper, follow these steps in order:
  1. Extract Logs: Unzip zuul_job_logs_2024_mark_filtered.zip into the project root.
  2. Restore Database: Use the mongorestore tool to import the study data from mongodb_archive.zip.
  3. Run Analysis: Execute scripts/BuildingResults.ipynb. This notebook contains all the logic to process the data and replicate the results, figures, and tables presented in the paper.
Note: scripts/detect_flaky_tests.py is the script used for daily data collection. You do not need to run this for replication, as the output is already provided in the  output/ folders.

Repository Structure

scripts/ – Data Collection and Analysis

  • BuildingResults.ipynb: The main Jupyter notebook used to replicate key results.
    • Note: This script is resource-intensive and may take a significant amount of time to complete.
  • detect_flaky_tests.py: The script used to collect flaky tests on a daily basis.

data/ – Dataset Files

  • flaky_tests.json: All flaky tests identified in the study.
  • cleaned_flaky_tests.json: Flaky tests that appeared in more than one patch set.
  • flaky_builds_with_inline_comments.json: Flaky builds annotated with inline comments.
  • flaky_jobs_with_inline_comments.json: Flaky jobs annotated with inline comments.
  • inconsistent_flakiness_sample.xlsx: Manually labeled sample used for RQ3 analysis.

output/

  • Output files generated by the detect_flaky_tests.py script.

materials/

  • Visualizations and figures used in the paper.

survey/

  • Questionnaire design and developer responses.

External Data Archives (stored separately)

  • zuul_job_logs_2024_mark_filtered.zip: A collection of job logs gathered during the study.
  • mongodb_archive.zip: A MongoDB database dump containing data used in this study.

Replication on the Zuul Ecosystem

  • zuul_replication.zip: Scripts and data to replicate our results on the Zuul ecosystem.

Files

replication_package.zip

Files (23.7 GB)

Name Size Download all
md5:4676e59b11e01f6287cbfa60c6ca3bcc
23.7 GB Preview Download