Seventeen provers under the hammer

This archive contains used benchmarks, raw results, scripts to create tables from the paper

and the provers used for all experiments in the the paper  Seventeen provers under the hammer.

The benchmarks used in the experiments are separated by the table in which they are used:

Note that there are 5000 benchmarks in each subclass of benchmarks.

Due to some technical issues we had to make sure our result collection

scripts ignored classes of 100 benchmarks. Which benchmarks are ignored

is specified below.

Similarly, the raw results are separated by the benchmarks they are evaluated on

The archives with results further contain raw results obtained from our

evaluation environment, StarExec. These files contain .csv files in which

each row describes one run of a prover on one benchmark and contains information

such as the used CPU time and memory resources, if the prover solved the problem and

path to the benchmark.

To create the tables used in paper from the raw results, extract all archives ending with _res in a

directory (let's label it <results>) and extract contents of table_scripts.zip in a directory labelled <scripts>.

Then, use the following commands to create the tables:

All provers, except for ENIGMA (which has several gigabyte installation) are

stored in provers.zip. Note that due to library dependencies,

and that they are packaged in StarExec-specific way,

the best way to install them is by reuploading them to StarExec.

ENIGMA is available at this link.