SAT-Inspired Eliminations for Superposition

This archive contains the problems, raw evaluation results and scripts for
running the experiments described in the paper "SAT-Inspired Inprocessing for
Superposition" by Petar Vukmirovic, Jasmin Blanchette, and Marijn J.H. Heule
available at

    https://matryoshka-project.github.io/pubs/satelimsup_paper.pdf

The problems we used are stored in the "problems/" subdirectory, together with
the required axioms. They are separated in two groups, "Theorems" and
"SatisfiableOrOpen", as in the paper.

In the "results/" directory, there are 5 files with names of the form
"figD[a|b].csv", where D is a digit from 1 to 5. The digit corresponds to the
figure from the paper, and the corresponding file contains experiment results
for the figure labeled D. The columns give information about the results of the
experiment run for a given prover configuration (e.g., CPU time, reported
status, memory usage). Each row corresponds to one problem file, whose name is
given in the "prob_name" column.

The "i_solver" column corresponds to a prover. The "i_configuration" column
corresponds to a configuration, where i is a natural number identifying a
prover-configuration combination.

Files named "fD[a|b]summary.csv" contain concise summaries of evaluation runs
for a corresponding "fD[a|b].csv" file. Their columns are of the form
"{solver}_{configuration}", and rows contain different statistics described in
the "summary" column.

The names of the configurations are self-explanatory and correspond to the ones
used in the paper. For BCE, SPE, and PPE, if the configuration has no
"inprocessing" in the name, the corresponding technique is used as
preprocessing.

The "zipperposition/" directory contains the scripts that execute the provided
Zipperposition binary (compiled only for Linux) on a given problem using a
given configuration. To run Zipperposition on a single problem using some
configuration, you can use scripts with the name "run_*.sh", where * stands for
the configuration used in the evaluation. The configuration names match the
ones described in the files "fD[a|b].csv".

The scripts take two arguments: (1) the path to the TPTP problem and (2) the
CPU timeout. For example, to run problem with the path "~/puzzling.p" using the
configuration "bce", execute

    ./run_bce.sh ~/puzzling.p 240

A working installation of python3 and bash is required.

The source code for Zipperposition can be obtained from the
"wip-pred-elim-hlt-congruence" branch of the Zipperposition git repository:
git@github.com:sneeuwballen/zipperposition.git. The binary stored in "scripts/"
corresponds to compiled sources tagged with the commit hash
c21457e0f0578a728a3779033bce28066bd85a2a. Compilation instructions are as in
the "README.md" file contained in the git repository.

Disclosure: Shortly before the CADE submission deadline, we noticed that there
is a bug with HLBE implementation that caused Zipperposition to wrongly reach
saturation on some unsatisfiable problems. This issue is now resolved in the
git repository (starting with 22194f27327568fc5183ff63f70d7fb6e362f7e3). Due to
lack of time, we could not redo the evaluation. For the evaluation on theorems,
the issue affected Zipperposition negatively for all HLBE-enabled modes, and we
expect to obtain better results with the new version. For the evaluation on
satisfiable or unknown problems HLBE was disabled, so the results are not
affected.