Evaluation Data and Scripts for SPLC 2025 Paper "Investigating the Effects of T-Wise Interaction Sampling for Vulnerability Discovery in Highly-Configurable Software Systems"

The evaluation data and scripts are structured into five distinct packages (available in the Files section found further down on this website):

1. data-and-results-axTLS: Vari-Joern report files for the analysis runs on axTLS and associated results created by executing the evaluation scripts (cf. scripts).
2. data-and-results-Fiasco: Vari-Joern report files for the analysis runs on Fiasco and associated results created by executing the evaluation scripts (cf. scripts).
3. data-and-results-BusyBox: Vari-Joern report files for the analysis runs on BusyBox and associated results created by executing the evaluation scripts (cf. scripts).
4. scripts: The Python evaluation scripts used to process the Vari-Joern reports and create the results that can then be plotted.
5. plotting: LaTeX source files used for plotting the results and the corresponding plots.

To reproduce the plots in the paper, execute the evaluation scripts (cf. scripts package) for the Vari-Joern analysis reports of the three subject systems (cf. data-and-results-axTLS, data-and-results-Fiasco, and  data-and-results-BusyBox). The LaTeX code leveraging tikz and pgfplots (cf. plotting package) can then be run with the final output of the evaluation scripts as input to create the plots. Refer to the README.md files located on the top level of every of the five packages for more information on the individual steps.

Example:

For reproducing Figure 4 (a) of the paper, one would have to:

1. Download the data-and-results-axTLS package.
2. Download the scripts package and execute the Postprocess.py evaluation script on the raw Vari-Joern reports found in data-and-results-axTLS (detailed instructions inside the README.md of the scripts package).
3. Execute the AnalyzeSampleSizesAndNumberOfFindings.py script of the script package on the output created by the Postprocess.py script.
4. Download the plotting package and execute the plot_vulnerability_warnings.tex file to create the plot in PDF format (detailed instructions inside the README.md of the plotting package).

Hint: For unzipping the individual packages, use the corresponding function of a file explorer (e.g., via right-click on the file --> Extract Here) or the command unzip <filename>.zip -d <destination_folder>.