Software Artifact for the CAV'19 Paper Titled "Overfitting in Synthesis: Theory and Practice"

Abstract:

In syntax-guided synthesis (SyGuS), a synthesizer's goal is to automatically generate a program belonging to a grammar of possible implementations that meets a logical specification. We investigate a common limitation across state-of-the-art SyGuS tools that perform counterexample-guided inductive synthesis (CEGIS). We empirically observe that as the expressiveness of the provided grammar increases, the performance of these tools degrades significantly.

We claim that this degradation is not only due to a larger search space, but also due to overfitting. We formally define this phenomenon and prove no-free-lunch theorems for SyGuS, which reveal a fundamental tradeoff between synthesizer performance and grammar expressiveness.

A standard approach to mitigate overfitting in machine learning is to run multiple learners with varying expressiveness in parallel. We demonstrate that this insight can immediately benefit existing SyGuS tools. We also propose a novel single-threaded technique called hybrid enumeration that interleaves different grammars and outperforms the winner of the 2018 SyGuS competition (Inv track), solving more problems and achieving a 5x mean speedup.

- - - - - - - -

The Artifact:

This software artifact contains all 180 invariant-inference SyGuS benchmarks, implementations of all 6 grammars presented in our paper, working code for the hybrid enumeration (HEnum) technique, and several scripts to reproduce the empirical claims made in the paper:

• Recording the number of failures with increasing grammar expressiveness,
• Measuring the running time and number of CEGIS rounds for LoopInvGen, and
• Comparing the performance of PLearn and HEnum

- - - - - - - -

Instructions:

1. Grab the cav19-artifact-100.zip file.
2. Optionally, verify that the sha1sum of the zip file matches with cav19-artifact-100.sha1:
   1. Download the cav19-artifact-100.zip and cav19-artifact-100.sha1 files to the same directory.
   2. In that directory, run: sha1sum -c cav19-artifact-100.sha1.
   3. You should see cav19-artifact-100.zip: OK.
3. Upon extracting the zip file, you should see:
   • CAV19_Artifact_100.ova VM image,
   • a copy of this file (Getting_Started.md), and
   • the artifact evaluation instructions (Instructions.html).
4. Download and install virtualbox on your machine.
5. Import the .ova image to your virtualbox.
6. Make sure you assign enough memory (8 GB minimum, 16 GB recommended).
7. Boot the system and perform some basic checks:
   1. Login with user cav and password ae.
   2. Start a terminal session (CTRL+ALT+T) and try:
      1. cd CAV_100
      2. make clean ; make dependencies
8. If you didn't notice any errors so far and you get the Everything built! message, then you can proceed with the artifact evaluation steps.