Static analysis evaluation experiment data

This repository contains the experiment artifacts for our paper entitled “Comprehensive Evaluation of Static Analysis Tools for Their Performance in Finding Vulnerabilities in Java Code” submitted to the IEEE Access Journal.

For each part of our experiment, we mention the related file name in this repository.

Research Methodology:

A. Experiment design: no related documents.

B. Preparing the Juliet Test Suite: 

In this section, we prepared Juliet for being analyzed by the five tools of the study. The related document for this section is called juliet_preparation.pdf.

C. Evaluation metrics: no related documents.

D. Experiment execution: 

Step 1: For each of the five tools, review the documentation to identify and activate the related checker(s).

The related documents of this section are pmd_checkers.pdf. spotbugs_fsb_checkers.pdf, infer_checkers.pdf, and sonar_checkers.pdf. Those documents include all the checkers that have been used and activated (if they were not active by default) to enable the Juliet analysis using the relevant checkers.

Step 2: Run each tool on each CWE and get the output reports.

The related document is called running_the_tools.pdf, which includes the detailed steps for running each tool.

Step 3: For each tool, and each CWE, consider the relevant checker's results.

no related document.

Step 4: For each tool, and each CWE, compute TP, FP, TN, and FN.

The related document is response_variables.xls

Step 5: Compute the response variables for each tool detecting each CWE.

The related document is response_variables.xls

Step 6: For each tool, compute collective evaluation metrics.

The related document is response_variables.xls