Ecosystem-Level Factors Affecting the Survival of Open-Source Projects: A Case Study of the PyPI Ecosystem - the dataset

Replication pack, FSE2018 submission #164:
------------------------------------------

**Working title:** Ecosystem-Level Factors Affecting the Survival of Open-Source Projects: 
A Case Study of the PyPI Ecosystem

**Note:** link to data artifacts is already included in the paper. 
Link to the code will be included in the Camera Ready version as well.


Content description
===================

- **ghd-0.1.0.zip** - the code archive. This code produces the dataset files 
  described below
- **settings.py** - settings template for the code archive.
- **dataset_minimal_Jan_2018.zip** - the minimally sufficient version of the dataset.
  This dataset only includes stats aggregated by the ecosystem (PyPI)
- **dataset_full_Jan_2018.tgz** - full version of the dataset, including project-level
  statistics. It is ~34Gb unpacked. This dataset still doesn't include PyPI packages
  themselves, which take around 2TB.
- **build_model.r, helpers.r** - R files to process the survival data 
   (`survival_data.csv` in **dataset_minimal_Jan_2018.zip**, 
   `common.cache/survival_data.pypi_2008_2017-12_6.csv` in 
   **dataset_full_Jan_2018.tgz**)
- **Interview protocol.pdf** - approximate protocol used for semistructured interviews.
- LICENSE - text of GPL v3, under which this dataset is published
- INSTALL.md - replication guide (~2 pages)

Replication guide
=================

Step 0 - prerequisites
----------------------

- Unix-compatible OS (Linux or OS X)
- Python interpreter (2.7 was used; Python 3 compatibility is highly likely)
- R 3.4 or higher (3.4.4 was used, 3.2 is known to be incompatible)

Depending on detalization level (see Step 2 for more details):
- up to 2Tb of disk space (see Step 2 detalization levels)
- at least 16Gb of RAM (64 preferable)
- few hours to few month of processing time

Step 1 - software
----------------

- unpack **ghd-0.1.0.zip**, or clone from gitlab:

      git clone https://gitlab.com/user2589/ghd.git
      git checkout 0.1.0
  
  `cd` into the extracted folder. 
  All commands below assume it as a current directory.
    
- copy `settings.py` into the extracted folder. Edit the file:
    * set `DATASET_PATH` to some newly created folder path
    * add at least one GitHub API token to `SCRAPER_GITHUB_API_TOKENS` 
- install docker. For Ubuntu Linux, the command is 
    `sudo apt-get install docker-compose`
- install libarchive and headers: `sudo apt-get install libarchive-dev`
- (optional) to replicate on NPM, install yajl: `sudo apt-get install yajl-tools`
  Without this dependency, you might get an error on the next step, 
  but it's safe to ignore.
- install Python libraries: `pip install --user -r requirements.txt` . 
- disable all APIs except GitHub (Bitbucket and Gitlab support were
  not yet implemented when this study was in progress): edit
  `scraper/init.py`, comment out everything except GitHub support
  in `PROVIDERS`.

Step 2 - obtaining the dataset
-----------------------------

The ultimate goal of this step is to get output of the Python function 
`common.utils.survival_data()` and save it into a CSV file:

    # copy and paste into a Python console
    from common import utils
    survival_data = utils.survival_data('pypi', '2008', smoothing=6)
    survival_data.to_csv('survival_data.csv')

Since full replication will take several months, here are some ways to speedup
the process:

####Option 2.a, difficulty level: easiest

Just use the precomputed data. Step 1 is not necessary under this scenario.

- extract **dataset_minimal_Jan_2018.zip**
- get `survival_data.csv`, go to the next step

####Option 2.b, difficulty level: easy

Use precomputed longitudinal feature values to build the final table.
The whole process will take 15..30 minutes.

- create a folder `<DATASET_PATH>/common.cache`, where `<DATASET_PATH>` is 
  the value of the variable `DATASET_PATH` in `settings.py`
- extract **dataset_minimal_Jan_2018** to the newly created folder
- rename files:

      mv backporting.csv monthly_data.pypi_backporting.csv
      mv cc_degree.csv monthly_data.pypi_cc_degree.csv
      mv commercial.csv monthly_data.pypi_commercial.csv
      mv commits.csv monthly_data.pypi_commits.csv
      mv contributors.csv monthly_data.pypi_contributors.csv
      mv dc_katz.csv monthly_data.pypi_dc_katz.csv
      mv downstreams.csv monthly_data.pypi_downstreams.csv
      mv d_upstreams.csv monthly_data.pypi_d_upstreams.csv
      mv github_user_info.csv user_info.pypi.csv
      mv issues.csv monthly_data.pypi_issues.csv
      mv non_dev_issues.csv monthly_data.pypi_non_dev_issues.csv
      mv non_dev_submitters.csv monthly_data.pypi_non_dev_submitters
      mv package_urls.csv package_urls.pypi.csv
      mv q90.csv monthly_data.pypi_q90.csv
      # raw_dependencies.csv is not required
      # raw_packages_info.csv is not required
      # Feel free to read README.md for more details about the data
      mv submitters.csv monthly_data.pypi_submitters.csv
      # In this scenario we'll generate a new survival_data.csv
      mv university.csv monthly_data.pypi_university.csv
      mv upstreams.csv monthly_data.pypi_upstreams.csv

- edit `common/decorators.py`, set `DEFAULT_EXPIRY` to some higher value,
  e.g. `DEFAULT_EXPIRY = float('inf')  # cache never expires`

Then, use the Python code above to obtain `survival_data.csv`.

####Option 2.c, difficulty level: medium

Use predownloaded raw data to build longitudinal feature values, 
and then the dataset. Despite most of the data is cached, some functions will
pull up updates which might take anywhere from days to couple weeks to run.


- Download **dataset_full_Jan_2018.tgz** from http://k.soberi.us/dataset_full_Jan_2018.tgz .
This file is not included in this archive because of its size 
(5.4Gb compressed, 34Gb unpacked).
- edit `common/decorators.py`, set `DEFAULT_EXPIRY` to some higher value,
  e.g. `DEFAULT_EXPIRY = float('inf')  # cache never expires`
- extract the content of this archive into `<DATASET_PATH>`.
- clean up `<DATASET_PATH>/common.cache` (otherwise you'll get Step 2.a. 
You can reproduce Step 2.b by deleting only `survival_data.pypi_2008_2017-12_6.csv`)

Run the Python code above to obtain `survival_data.csv`.

####Option 2.d, difficulty level: hard

Build the dataset from scratch. Although most of the processing is
parallelized, it will take at least couple months on a reasonably
powerful server (32 cores, 512G of RAM, 2Tb+ of HDD space in our setup).

- ensure the `<DATASET_PATH>` is empty
- add more GitHub tokens (borrow from your coworkers) to `settings.py`.

Run the Python code above to obtain `survival_data.csv`.


Step 3 - run the regression
---------------------------

install R libraries:

    install.packages(c("htmlTable", "OIsurv", "survival", "car", "survminer", 
                       "ggplot2", "sqldf", "pscl", "texreg", "xtable"))

Use `build_model.r` (e.g. in RStudio) and produced `survival_data.csv` to 
build the regressions used in the paper. This process takes at least 16Gb of RAM 
and takes few hours to run due to the gigantic size of the dataset.