Does Unit-Tested Code Crash? A Case Study of Eclipse: Replication Package

Does Unit-Tested Code Crash? A Case Study of Eclipse: Replication Package

This is a replication package associated with the paper titled “Does Unit-Tested Code Crash? A Case Study of Eclipse”. Below is a description of the package’s contents.

Data

Data files associated with the paper are provided in the data directory.

Text file tested-crashed.txt

Data specifying whether methods were tested and whether they crashed (according to the criteria adopted in the study). Extracted from matches.xlsx. The data are used as input for Fisher’s test (RQ1).

Spreadsheet matches.xlsx

Test coverage data and calculations associated with failed methods, class coverage, and matching method coverage results are provided in an Excel spreadsheet. Below is the description of the spreadsheet’s contents.

Worksheet Test Coverage

Contains the data regarding the JaCoCo test code coverage analysis.

• Class: The name of the class in which a method appears in JVM internal form notation
• Method: The method’s name
• Parameters: The method’s arguments in JVM parameter descriptor format; required to handle Java’s {} polymporphism
• Class Has Unit Test: Whether the corresponding class has associated unit test code
• Class Unit-Test Line Density: The ratio of lines in class’s test code over those in the class’s implementation code
• Covered Instructions / Branches / Lines: As reported by JaCoCo
• Total Instructions / Branches / Lines: As reported by JaCoCo
• Covered Instructions / Branches / Lines ratio: The ratio between the two preceding values; 1 for methods without any branches
• Top-1 / Top-6 / Top-10 : In how many stack traces the method appears within; the top-10 / top-6 / the very first stack frame(s)
• Tested: TRUE if the method is considered tested by having a test code coverage above the median (0.966) and an associated test class
• Crashed: TRUE if the method has crashed as evidenced by its appearance on the topmost stack frame
• Stack trace file names: in which the method appeared

Worksheet Test Existence

Contains the data of the analysis regarding the existence of test code.

• Class: Class containing implementation code
• TestClassNames: Classes that contain tests for the above
• Number of relevant tests
• Lines in class test code
• Lines of class
• Class Unit-Test Line Density: The ratio between the two above

Worksheet Metrics

Contains the derivation of metrics reported in the paper. In the cases of tables these are formatted in LaTeX for direct incorporation into the text.

Spreadsheet jacoco.xlsx

Complete test coverage data obtained from JaCoCo are provided in an Excel spreadsheet. Below is the description of the spreadsheet’s contents.

Worksheet Data

Contains the following method code coverage fields as reported by JaCoCo, as well as the calculated percentages.

• Class
• Method
• Parameters
• Covered Instructions
• Total Instructions
• % Covered Instructions
• Covered Branches
• Total Branches
• % Covered Branches
• Covered Lines
• Total Lines
• % Covered Lines

Worksheet Metrics

Contains the derivation of numbers reported in the preliminary quantitative analysis and Figure 2.

Compressed tar archive eclipse-src.tar.gz

Contains the Eclipse source code used for running the Eclipse tests with JaCoCo code coverage analysis. It was obtained from the Eclipse source code repositories as follows.

• Clone the Eclipse aggreagator repository into a directory named z by running: git clone -b master --recursive git://git.eclipse.org/gitroot/platform/eclipse.platform.releng.aggregator.git z
• In the z directory, checking out the used release by running cd z && git submodule foreach git checkout M20160212-1500
• Checking out the release for the main repository by running: git checkout M20160212-1500
• Applying the patch eclipse-src.diff

Patch file eclipse-src.diff

See above.

Zip file incidents.zip

Contains the 126,026 incidents (crash report stack traces and meta-data) associated with EclipseProduct org.eclipse.epp.package.java.product and BuildID 4.5.2.M20160212-1500. This is a subset from the two million incidents available as the AERI stack traces data set.

The subset of incidents was extracted from the full AERI data set with the following command.

for f in *; do
  grep -q org.eclipse.epp.package.java.product $f && grep -q 4.5.2.M20160212-1500 $f && mv $f selected-files/
done

Compressed file jacoco.xml.gz

Contains the results of the JaCoCo code coverage analysis over the Eclipse testing.

Code

The following scripts are provided in the src directory

• extract.py: script for extracting crash (incidents) and coverage (JaCoCo) data
• unit-tested-classes.py: script for finding the classes with associated unit test code
• merge.py: script for matching crash (incidents) with coverage (JaCoCo) data
• fisher.r: R script for running Fisher’s test