UPDATE: Zenodo migration postponed to Oct 13 from 06:00-08:00 UTC. Read the announcement.
There is a newer version of this record available.

Dataset Open Access

Ecosystem-Level Factors Affecting the Survival of Open-Source Projects: A Case Study of the PyPI Ecosystem - the dataset

Valiev, Marat; Vasilescu, Bogdan; Herbsleb, James

Replication pack, FSE2018 submission #164:
**Working title:** Ecosystem-Level Factors Affecting the Survival of Open-Source Projects: 
A Case Study of the PyPI Ecosystem

**Note:** link to data artifacts is already included in the paper. 
Link to the code will be included in the Camera Ready version as well.

Content description

- **ghd-0.1.0.zip** - the code archive. This code produces the dataset files 
  described below
- **settings.py** - settings template for the code archive.
- **dataset_minimal_Jan_2018.zip** - the minimally sufficient version of the dataset.
  This dataset only includes stats aggregated by the ecosystem (PyPI)
- **dataset_full_Jan_2018.tgz** - full version of the dataset, including project-level
  statistics. It is ~34Gb unpacked. This dataset still doesn't include PyPI packages
  themselves, which take around 2TB.
- **build_model.r, helpers.r** - R files to process the survival data 
   (`survival_data.csv` in **dataset_minimal_Jan_2018.zip**, 
   `common.cache/survival_data.pypi_2008_2017-12_6.csv` in 
- **Interview protocol.pdf** - approximate protocol used for semistructured interviews.
- LICENSE - text of GPL v3, under which this dataset is published
- INSTALL.md - replication guide (~2 pages)
Replication guide

Step 0 - prerequisites

- Unix-compatible OS (Linux or OS X)
- Python interpreter (2.7 was used; Python 3 compatibility is highly likely)
- R 3.4 or higher (3.4.4 was used, 3.2 is known to be incompatible)

Depending on detalization level (see Step 2 for more details):
- up to 2Tb of disk space (see Step 2 detalization levels)
- at least 16Gb of RAM (64 preferable)
- few hours to few month of processing time

Step 1 - software

- unpack **ghd-0.1.0.zip**, or clone from gitlab:

      git clone https://gitlab.com/user2589/ghd.git
      git checkout 0.1.0
  `cd` into the extracted folder. 
  All commands below assume it as a current directory.
- copy `settings.py` into the extracted folder. Edit the file:
    * set `DATASET_PATH` to some newly created folder path
    * add at least one GitHub API token to `SCRAPER_GITHUB_API_TOKENS` 
- install docker. For Ubuntu Linux, the command is 
    `sudo apt-get install docker-compose`
- install libarchive and headers: `sudo apt-get install libarchive-dev`
- (optional) to replicate on NPM, install yajl: `sudo apt-get install yajl-tools`
  Without this dependency, you might get an error on the next step, 
  but it's safe to ignore.
- install Python libraries: `pip install --user -r requirements.txt` . 
- disable all APIs except GitHub (Bitbucket and Gitlab support were
  not yet implemented when this study was in progress): edit
  `scraper/init.py`, comment out everything except GitHub support

Step 2 - obtaining the dataset

The ultimate goal of this step is to get output of the Python function 
`common.utils.survival_data()` and save it into a CSV file:

    # copy and paste into a Python console
    from common import utils
    survival_data = utils.survival_data('pypi', '2008', smoothing=6)

Since full replication will take several months, here are some ways to speedup
the process:

####Option 2.a, difficulty level: easiest

Just use the precomputed data. Step 1 is not necessary under this scenario.

- extract **dataset_minimal_Jan_2018.zip**
- get `survival_data.csv`, go to the next step

####Option 2.b, difficulty level: easy

Use precomputed longitudinal feature values to build the final table.
The whole process will take 15..30 minutes.

- create a folder `<DATASET_PATH>/common.cache`, where `<DATASET_PATH>` is 
  the value of the variable `DATASET_PATH` in `settings.py`
- extract **dataset_minimal_Jan_2018** to the newly created folder
- rename files:

      mv backporting.csv monthly_data.pypi_backporting.csv
      mv cc_degree.csv monthly_data.pypi_cc_degree.csv
      mv commercial.csv monthly_data.pypi_commercial.csv
      mv commits.csv monthly_data.pypi_commits.csv
      mv contributors.csv monthly_data.pypi_contributors.csv
      mv dc_katz.csv monthly_data.pypi_dc_katz.csv
      mv downstreams.csv monthly_data.pypi_downstreams.csv
      mv d_upstreams.csv monthly_data.pypi_d_upstreams.csv
      mv github_user_info.csv user_info.pypi.csv
      mv issues.csv monthly_data.pypi_issues.csv
      mv non_dev_issues.csv monthly_data.pypi_non_dev_issues.csv
      mv non_dev_submitters.csv monthly_data.pypi_non_dev_submitters
      mv package_urls.csv package_urls.pypi.csv
      mv q90.csv monthly_data.pypi_q90.csv
      # raw_dependencies.csv is not required
      # raw_packages_info.csv is not required
      # Feel free to read README.md for more details about the data
      mv submitters.csv monthly_data.pypi_submitters.csv
      # In this scenario we'll generate a new survival_data.csv
      mv university.csv monthly_data.pypi_university.csv
      mv upstreams.csv monthly_data.pypi_upstreams.csv

- edit `common/decorators.py`, set `DEFAULT_EXPIRY` to some higher value,
  e.g. `DEFAULT_EXPIRY = float('inf')  # cache never expires`

Then, use the Python code above to obtain `survival_data.csv`.

####Option 2.c, difficulty level: medium

Use predownloaded raw data to build longitudinal feature values, 
and then the dataset. Despite most of the data is cached, some functions will
pull up updates which might take anywhere from days to couple weeks to run.

- Download **dataset_full_Jan_2018.tgz** from http://k.soberi.us/dataset_full_Jan_2018.tgz .
This file is not included in this archive because of its size 
(5.4Gb compressed, 34Gb unpacked).
- edit `common/decorators.py`, set `DEFAULT_EXPIRY` to some higher value,
  e.g. `DEFAULT_EXPIRY = float('inf')  # cache never expires`
- extract the content of this archive into `<DATASET_PATH>`.
- clean up `<DATASET_PATH>/common.cache` (otherwise you'll get Step 2.a. 
You can reproduce Step 2.b by deleting only `survival_data.pypi_2008_2017-12_6.csv`)

Run the Python code above to obtain `survival_data.csv`.

####Option 2.d, difficulty level: hard

Build the dataset from scratch. Although most of the processing is
parallelized, it will take at least couple months on a reasonably
powerful server (32 cores, 512G of RAM, 2Tb+ of HDD space in our setup).

- ensure the `<DATASET_PATH>` is empty
- add more GitHub tokens (borrow from your coworkers) to `settings.py`.

Run the Python code above to obtain `survival_data.csv`.

Step 3 - run the regression

install R libraries:

    install.packages(c("htmlTable", "OIsurv", "survival", "car", "survminer", 
                       "ggplot2", "sqldf", "pscl", "texreg", "xtable"))

Use `build_model.r` (e.g. in RStudio) and produced `survival_data.csv` to 
build the regressions used in the paper. This process takes at least 16Gb of RAM 
and takes few hours to run due to the gigantic size of the dataset.
Files (5.8 GB)
Name Size
19.5 kB Download
5.8 GB Download
30.9 MB Download
981.5 kB Download
6.1 kB Download
5.8 kB Download
Interview protocol.pdf
60.8 kB Download
35.1 kB Download
1.6 kB Download
1.1 kB Download
962 Bytes Download
All versions This version
Views 990795
Downloads 984438
Data volume 1.9 TB434.8 GB
Unique views 892739
Unique downloads 402204


Cite as