Artifacts for [On Rank Aggregating Test Prioritizations]

Artifacts for [On Rank Aggregating Test Prioritizations]
=====================================================

username: entp
password: entp
root password: entp

Directory structure for [/home/entp/EnTP]
-----------------------------------------------------------------
[benchmarks] -> contains the projects/subjects under test used in our study.

[cost_cov_diff] -> pre-recorded cost (cachegrind's I-Ref count) per test-case. We recorded this precporcessing to avoid fluctuations in costs across different systems, and hence maintain uniformity.

[raw_data_scripts] -> contains C++ implementation of EnTP, and scripts to generate results (tables, boxplots, .csv, .txt, etc.)

[exp_res_raw] -> contains pre-recorded experiments results (as .csv files) as reported in the paper. To generate the plots, run [python3 plot.py] from {/home/entp/EnTP/exp_res_raw}. The plots will be generated in .eps format. These precomputed results helps the artifact reviewer save time by generating the plots only, otherwise the whole process (detailed in Makefile's usage) generates a [database] directory of ~400GB (all benchmarks, all combinations, all results even beyond those reported in the paper, + some extra logs) which is very time consuming (~1 month on a standard laptop with 8 cores, 8GB RAM).

Makefile's usage
---------------------------
[Step 1]

entp@entp:~/EnTP$ make -s entp_all_[benchmark]

Possible values of [benchmark] = {c4, gravity, mlisp, replace, schedule2, space, xc, cf, grep, printtokens, scd, sed, tcas, xxhash, flex, gzip, printtokens2, schedule, slre, totinfo}.

example:

entp@entp:~/EnTP$ make -s entp_all_slre

executes EnTP and state-of-the-arts on the benchmark "slre". Please follow log messages displayed after executing the above command. At the end of the execution, a sub-diretory named [database] will store the results for all experiments performed on "slre" for the current system and environment.
...
[database/{100, 75, 50, 25}] -> contains experimental results for the consensus budget of top-{100%, 75%, 50%, 25%}.
...

(optional)

entp@entp:~/EnTP$ make -s entp_all_[benchmark] #other benchmarks

[Step 2]

entp@entp:~/EnTP$ make -s generate_tabs

This will collect data from the newly generated directories and results (at the end of previous step), and generate tables, and .eps plots.

Check for results reproduced
---------------------------------------------
You can visually compare the .eps plots in {/home/entp/EnTP} with the ones generated in the directory {/home/entp/EnTP/exp_res_raw}.
You can also compare the .csv files under these directories for quantitative comparison with some tolerance.

(optional)

entp@entp:~/EnTP$ make -s destroy_all

Cleans up everything!