Artifacts of Tactic Learning and Proving for the Coq Proof Assistant

Two artifacts are accompanying the paper "Tactic Learning and Proving for the Coq Proof Assistant". A dataset, and a set of configurations of the software that generated the dataset.

1. Dataset

The dataset that was created for the Standard Library evaluation can found in dataset.tar.7z.

1.1 Offline Evaluation

The folder offline-eval/stdlib-feats/ contains a textual representation of the tactic database built during the compilation of the standard library. For every recorded tactic in every file, a line exists with a list of integer features of the proof state and a hash of the corresponding tactic.

The folder stdlib-eval-offline/theories contains, for every file, an evaluation of the different \(k\)-NN predictors, with or without intra-lemma learning and with or without file dependencies. Every line represents an attempt to predict the correct tactic from the database. The number represents the index of the predicted list at which the correct tactic was found. Index \(-1\) means the tactic was not found.

The folder stdlib-eval-offline/aggregated/ contains aggregated versions of the files described above, where all development files are concatenated. Finally, stdlib-eval-offline/cumulative/ contains the fraction of tactics that can be predicted by the \(k\)-NN algorithm for every \(k\).

1.2 Search Evaluation

The folder search-eval/ contains for every configuration of the system presented in the paper an evaluation of the lemmas in the standard library that can be proven automatically. For every development file, there is a corresponding file with a list of its lemmas. If the lemma was proven, the successful trace of the search procedure is shown.

2. Executable configurations of the system

All configurations of our system that are presented in the paper can be found in coq-configuration-sources.tar.7z. Each configuration is a modification of the Coq codebase. The code is only tested on Linux. To build the code, similar system requirements as the normal Coq build are needed:

• A recent version of OCaml, versions 4.07.0 to 4.09.0 are tested
• Optionally, the Opam package manager
• The Ocamlfind library
• The Num library

For people unfamiliar with OCaml and Opam follows a short installation guide for Linux. First, install Opam 2.x through your favorite package manager. Some package managers do not yet include Opam 2.0. For Ubuntu 18.04 a custom ppa is available. On other systems, you can use a binary installation script. After installation, execute the following commands.

opam init # answer yes to any questions the initialization script may ask
eval $(opam env)
opam switch create 4.09.0
eval $(opam env)
opam install ocamlfind num

The system should now be ready to build the configurations.

The sources of the configurations can be used to recreate the search evaluation files and feature files in the dataset described above. To do this for a particular configuration, execute the following commands in the main directory of that configuration.

./configure -local
make benchmark -j{n}

Here {n} is the amount of cores to dedicate to this pursuit (we recommend as many as possible to prevent boredom). Note that warnings and errors are expected during the build. If the build gets aborted, trouble-shooting follows the same process as building the standard version of the Coq codebase. For this, we refer you to Coq's INSTALL.md and INSTALL.make.md. After the build finishes, the evaluation files should appear amongst the source code.