On the rise and fall of CI services in GitHub

# Content of this repository

This repository contains the replication package for the article "On the rise and fall of CI services in GitHub" co-authored by Mehdi Golzadeh, Alexandre Decan and Tom Mens, and published in the  IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER 2022).

The file "requirements.txt" contains a list of dependencies needed to execute the scripts, and can be installed as normal using "pip install -r requirements.txt". Before using the notebooks, it is strongly recommended that you create an isolated environment with Python 3.6. This can be done easily using 'pew' or 'virtualenv'.

pip install pew
pew new -p python3.6 -r requirements.txt CIstudy

or

pip install virtualenv
virtualenv -p python3.6 CIstudy

The scripts can be found under the "Notebooks" directory and they must be run with 'Jupyter notebook'. Notebook "_A_Data_Cleaning" uses 2 raw datasets of "dataset_raw_700k_allCIs.csv.gz" and "all_activerepos_notfork676K.csv.gz" and generates "NPM_cs_protions_700k_7cis" dataset for "_B_Analysing CI landscape in NPM". All datasets are provided in the data directory.

We used (https://replicate.npmjs.com/_all_docs) endpoint to be able to get the list of NPM packages and their info. The info that we managed to obtain from the API includes:

Name: the package name
Created: time of the creation
Updated: last update time recorded for the package
Maintainers_name: name of the maintainer
Maintainers_email: email address of the maintainer
Homepage: home page
Repository: repository address of the package
Author: author name
License: license type

Out of the 1601989 packages listed in the packages list, we managed to access and clone 676K of them. Using git commands, we extracted files from the entire history of git, then iterated over all commits, and found where CI files were added or deleted. In order to determine which repositories are forked and which are not, we used the GitHub API.

Travis CI service references

Here are a few examples of (negative) reactions caused by Travis' decision to change its pricing model for open-source projects. These references accompany section "VII: Discussion" of the SANER2022 paper:

https://travis-ci.community/t/org-com-migration-unexpectedly-comes-with-a-plan-change-for-oss-what-exactly-is-the-new-deal/10567
https://medium.com/dictcp/how-travis-ci-kills-the-productivity-of-a-100-team-in-2-hours-9e66ed275f5a
https://www.jeffgeerling.com/blog/2020/travis-cis-new-pricing-plan-threw-wrench-my-open-source-works
https://travis-ci.community/t/contradictory-information-in-the-ui-on-whether-credits-are-replenished-on-the-free-plan/10518
https://news.ycombinator.com/item?id=25338983
https://earthly.dev/blog/migrating-from-travis/
https://www.getapp.com/development-tools-software/a/travis-ci/pricing/
https://discourse.julialang.org/t/travis-ci-changes-pricing-plans/49528/15
https://github.com/openstack4j/openstack4j/issues/165
https://github.com/iridia-ulb/references/issues/10
https://github.com/neopoly/two_thousand_forty_eight/pull/1
https://github.com/tueda/homebrew-loops/issues/18
https://github.com/kobotoolbox/kpi/issues/2898
https://github.com/MTG/essentia.js/issues/63
https://github.com/project-generator/project_generator/issues/490
https://github.com/JULIELab/jcore-base/issues/122
https://github.com/RaRe-Technologies/gensim-wheels/issues/15
https://github.com/opensourcepos/opensourcepos/issues/2834
https://github.com/reviewdog/reviewdog/issues/939
https://github.com/Trustroots/trustroots/issues/2389
https://github.com/cloudflare/cfssl/issues/1194
https://github.com/lobsters/lobsters/issues/1011
https://github.com/mattn/goveralls/issues/194
https://github.com/JonathonReinhart/staticx/issues/124
https://github.com/opentripplanner/OpenTripPlanner/pull/3303
https://github.com/audreyfeldroy/cookiecutter-pypackage/issues/622
https://github.com/logrotate/logrotate/issues/405
https://github.com/florafauna/ReplicationSuccess/issues/56
https://github.com/sympy/sympy/issues/20374
https://github.com/Unvanquished/Unvanquished/issues/1432
https://github.com/Cyan4973/xxHash/issues/529
https://github.com/xsf/infrastructure/issues/13
https://github.com/impega/reciptacle/issues/7
https://github.com/Rust-GCC/gccrs/issues/477
https://github.com/agda/agda/issues/5061
https://github.com/Toblerity/Fiona/issues/975
https://github.com/gsauthof/dracut-sshd/issues/43
https://github.com/aio-libs/aiomysql/issues/541
https://github.com/vorburger/MariaDB4j/issues/451
https://github.com/coriolinus/lerp-rs/issues/7
https://github.com/markosamuli/ansible-pyenv/issues/45
https://github.com/microformats/php-mf2/issues/224
https://github.com/soft-matter/pims/issues/393
https://github.com/clawpack/clawpack/issues/194
https://github.com/eclipse/sw360/issues/985
https://github.com/airr-community/airr-standards/issues/484
https://github.com/docker-science/cookiecutter-docker-science/issues/107
https://github.com/lh3/minimap2/issues/740
https://github.com/SADevs/sadevs.github.io/issues/64
https://github.com/hyphacoop/two.compost.digital/issues/69

Limitations in the AppVeyor CI service

Here are a few examples of issues raised by software developers related to limitations in the AppVeyor CI service. These references accompany section "VII: Discussion" of the SANER2022 paper:

https://help.appveyor.com/discussions/questions/41921-appveyor-builds-more-than-twice-as-slow-as-travis
https://github.com/python/mypy/issues/8292
https://github.com/torquem-ch/silkworm/issues/358
https://github.com/quantopian/zipline/issues/2619
https://github.com/henninglive/logitech-lcd/issues/12
https://github.com/wheybags/freeablo/issues/339
https://github.com/Framstag/libosmscout/issues/1034
https://github.com/PlasmaPy/SpectroscoPyx/issues/53
https://github.com/apiaryio/dredd/pull/1804
https://github.com/royaltm/node-murmurhash-native/issues/16
https://githubmemory.com/repo/Framstag/libosmscout/issues/1034