# Evaluation Artifact for VariantInc: Automatically Pruning and Integrating Versioned Software Variants

## Structure

This artifact consists of multiple parts. It contains the raw data reported in the paper, the VariantInc tool and evaluation setup, including all its sources and its build system, and the scripts for generating the plots shown in the paper. The root directory contains the two sub directories _data_ and _tool_.

### Data
- _data_ contains the raw data and plots for the original evaluation of VariantInc
  - In _data/statistics_ are two .csv files per analyzed system
    - The file _<system_name>.tree.csv_ contains information about the version history, including the number of forks, branches, and commits
    - The file _<system_name>.other.csv_ contains information about the contents of a system, such as number of files, number of lines, and number of variants
- _data/plots_ contains generated files, including plots and tables

### Tool

- _tool_ contains the source files and executable of VariantInc and all scripts for analyzing the generated data
- VariantInc is part of the tool [FeatJAR](https://github.com/FeatureIDE/FeatJAR), which is a tool for feature-oriented software development based on [FeatureIDE](https://github.com/orgs/FeatureIDE/repositories?q=FeatureIDE)

---
## Usage

- The following step show how to clone, build, and run VariantInc and how to analyze its results
- Any of these steps can be skipped as this artifact includes all built and generated files

### Clone

- Clone the root project
```git clone https://github.com/FeatureIDE/FeatJAR.git --branch evaluation\_variant\_inc tool```
- Change into the cloned directory
```cd tool```
- Clone all sub projects
```./scripts/clone.bat```

### Build

- Change into the variant-inc project
```cd tool/evaluation-variant-inc```
- Run gradle
```./gradlew clean assemble```

This created the file \texttt{tool/evaluation-variant-inc/build/libs/evaluation-variant-inc-0.1.0-SNAPSHOT -all.jar}, which is a standalone executable JAR file for running the evaluation of VariantInc.

### Run

- Change into the variant-inc project
```cd tool/evaluation-variant-inc```
- Execute the JAR file
```java -Xmx12g -jar build/libs/evaluation-variant-inc-0.1.0-SNAPSHOT-all.jar merge-variants systems\_test.list -log=2```
  - You may want to change the parameter _-Xmx_ depending on the memory available on your machine
  - The parameter _-log_ can be set to 0 for substantially less console output.

The last parameter provides a file containing the list of repositories to analyze. This artifact includes three default lists:
- _systems_test.list_: includes just a single system for testing purposes
- _systems_sublist.list_: includes a small subset of all evaluated systems
- _systems.list_: includes all evaluated systems

We recommend that the first run uses _systems_test.list_, because processing all systems takes in a considerable amount of time (depending on the used hardware and internet connection). In addition, processing many systems at once may exceed Github's daily limit of API usage for unregistered users. During the execution, you are asked for a Github user name and password, you can simply press the enter key twice to skip this and continue as an unregistered user.

The generated files are stored within the newly created directory _tool/evaluation-variant-inc/gen_.
Within this directory there are four sub directories:
- _auxiliary_ contains the configurable VariantInc platform for each system, integrating all commits from each fork and branch
- _logs_ contains the console output for each run.
- _repositories_ contains the cloned repositories for each system.
- _statistics_ contains the .csv files with statistical data about the generated VariantInc platform.

### Analyze


- Install required python and R libraries (**Warning**: this may modify your system)
```./tool/evaluation-variant-inc/eval\_scripts/setup\_r\_and\_python.sh```
- Run the python script
```python tool/evaluation-variant-inc/eval\_scripts/eval.py tool/evaluation-variant-inc/gen```
- Run the R script
```Rscript tool/evaluation-variant-inc/eval\_scripts/eval.R tool/evaluation-variant-inc/gen```

The last parameter for the python and R script determines the location of the input and output files. To analyze the original files provided with this artifact use _data_ instead of _tool/evaluation-variant-inc/gen_. Within the provided location the python script creates the directory _plots_.
