Results of the CT competition at the 12th International Workshop on Combinatorial Testing

These are the models and raw results achieved by participating tools in the CT competition at the 12th International Workshop on Combinatorial Testing (IWCT 2023).

The file ct-comp-iwct2023-models.tar.gz contains the input files in CTWedge and ACTS format.

The file ct-comp-iwct2023-output-and-timing.tar.gz contains one folder per tool and strength. In each of these folders, files conform to the following syntax:[toolname]_[model_filename]_[strength]_[iteration].[suffix]
where "toolname" is identical to the current folder,
"model_filename" is the exact name of the model file (including suffix),
"strength" is the combinatorial strength of the resulting CA (between 2 and 5),
"iteration" takes values between 1 and 3 and refers to the round of execution (at lower strengths, tools were tested using 3 iterations; at higher strengths, some tools required too much execution time and were only run once),
"suffix" is one of the following: "time" for the output of GNU time in verbose mode or "out" for the resulting CA (in headerless CSV form).

The competing tools are (in alphabetical order):

• ACTS
• CAgen (in three versions; the default uses one thread less than the number of cores, _n uses exactly the number of available cores, and _np1 uses one additional thread)
• caopt
• KALI
• MEDICI
• pMEDICI

The file validation-results.zip contains the results of the CA validation of all output files.

All tools were executed in Docker containers based on the official OpenJDK image, which is itself based on Debian Buster (Docker identifier: openjdk:14-buster). The experiments were performed on a machine using a Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (16 physical cores, 32 logical cores) with 256 GB RAM.