Replication Kit: "On the Defect-Detection Capabilities of Unit and Integration Tests"

Replication Kit for the Paper "On the Defect-Detection Capabilities of Unit and Integration Tests"
This additional material shall provide other researchers with the ability to replicate our results. Furthermore, we want to facilitate further insights that might be generated based on our data sets.

Structure
The stucture of the replication kit is as follows:

• additional_visualizations: contains additional visualizations (Venn-Diagrams) for each projects for each of the data sets that we used
• data_analysis: contains two python scripts that we used to analyze our raw data (one for each research question)
• data_collection_tools: contains all source code used for the data collection, including the used versions of the COMFORT framework, the BugFixClassifier, the script that was used to filter and collect the issues with their commits, and the used tools of the SmartSHARK environment
• mongodb_no_authors: Archived dump of our MongoDB that we created by executing our data collection tools. The "comfort" database can be restored via the mongorestore command.
• project_defects.csv: Raw data of our manual data collection process. It included all collected defects including the name of the issue, its description and the commit references in which these issues were fixed. Furthermore, it includes information about which test failed or errored after re-integrating the defect into the analyzed release version of the project.

Additional Visualizations
We provide three additional visualizations for each project:

1. Disjoint-Mutation-Data (visualizations for the DISJ data set)
2. Mutation-Data (visualizations for the ALL data set)
3. Seeded-Data (visualizations for the SEEDED data set)

For each of these data sets there exist one visualization for each project that shows six Venn-Diagrams, where five of them present the different defect types and one overall Venn-Diagram. These Venn-Diagrams show the number of defects that were detected by either unit, or integration tests (or both).

Analysis scripts
Requirements:

• python3.5
• tabulate
• scipy
• seaborn
• mongoengine
• pycoshark
• pandas
• matplotlib

Both python files contain all code for the statistical analysis we performed.

Data Collection Tools
We provide all data collection tools that we have implemented and used throughout our paper. Overall it contains six different projects and one python script:

• BugFixClassifier: Used to classify our defects.
• comfort-core: Core of the comfort framework. Used to classify our tests into unit and integration tests and calculate different metrics for  these tests.
• comfort-jacoco-listner: Used to intercept the coverage collection process as we were executing the tests of our case study projects.
• filter_issues.py: Used to filter and collect issues with their commits (need the vcsSHARK and issueSHARK executed beforehand)
• issueSHARK: Used to collect data from the ITSs of the projects.
• pycoSHARK: Library that contains models for the used ORM mapper that is used insight the SmartSHARK environment.
• vcsSHARK: Used to collect data from the VCSs of the projects.