Online appendix to State of the CArt: Evaluating Covering Array generators at scale

This dataset includes the input models, figures, and resulting output of all CA generation tools evaluated in the context of the article "State of the CArt: Evaluating Covering Array generators at scale".

The file `models.tar.gz` includes the 295 input parameter models in four different dialects (ACTS, CTWedge/pMEDICI, PICT, and CASA).

The file `figures.zip` contains the underlying data of the figures included in the article and the resulting figures (as well as a number of additional visualizations that were not used in the published work).

The files beginning with `evaluation` contain the output, timing, and error messages produced by each of the competing tools, in one folder per strength (named after a timestamp in ascending order of strength).
In each of these folders, files conform to the following syntax:[toolname]_[model_filename]_[strength]_[iteration].[suffix]
where "toolname" is identical to the current folder,
"model_filename" is the exact name of the model file (including suffix),
"strength" is the combinatorial strength of the resulting CA (between 2 and 5),
"iteration" takes values between 1 and 3 and refers to the round of execution (at lower strengths, tools were tested using 3 iterations; at higher strengths, some tools required too much execution time and were only run once),
"suffix" is one of the following: "time" for the output of GNU time in verbose mode; "out" for the resulting CA (in headerless CSV form); "err" for the standard output (usually empty). Note that additional intermediate files may exist for some tools (e.g. when their original output is not strictly a headerless CSV).

All tools were executed in Docker containers based on the official OpenJDK image, which is itself based on Debian Buster (Docker identifier: openjdk:14-buster). The experiments were performed on a machine using a Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (16 physical cores, 32 logical cores) with 256 GB RAM.

CASAConverterCLI.jar is a standalone Java program that converts CTWedge models to CASA models. It accepts the path to a CTWedge model and the strength as its parameters.

ct-eval-framework.tar.bz2 contains the evaluation framework itself. It can be used to replicate our results and evaluate your own CA generator. Setup instructions are provided in the README.md file wihtin the archive.

wca-gcc-11-fix.patch is a patch that enables the compilation of WCA under newer versions of GCC.