Programming with a Read-Eval-Synth Loop (Artifact)

Quick setup guide for evaluating RESL submission and understanding its components.

Getting started:

Required Software:

VM Login:

username: resl password: reslresl

Kicking the tires:

  1. Import and run VM.
  2. Quick-test the empirical benchmarks:
    1. Open an interactive shell.
    2. Run bench. This will run 4 benchmarks from each of the empirical benchmark sets, and will take approximately 15 minutes.
    3. If all runs succeed, "OK!" will be printed.
  3. Quick-test the RESL tool:
    1. Open an interactive shell.
    2. Run command resl to start the synthesis engine.
    3. this will automatically open the browser at http://localhost:3000.
    4. To try the overview example in the paper:
      1. Enter examples
        • Input: 420, Output: ['4','42','420']
        • Input: 2020, Output: ['2','20','202','2020']
      2. Enter the program [1,input.toString().length].map(i => input.toString().slice(0,i)) in the text bar at the bottom and hit enter.
      3. Hover over [1,input.toString().length], right-click, and choose "fix this" to make a hole.
      4. Hover over input.toString().length, right click and choose "retain" to retain this subexpression.
      5. Click the Synthesize button at the bottom right corner, and wait for the result to be displayed.

Reproducing Experiments

Paper claims vs. artifact claims

List of claims in the paper supported by the artifact:

  1. Effects of predicate type on the OE-reduction (Section 8.1)
  2. Necessity of OE on the completion (Section 8.2)

List of claims not supported by the artifact:

  1. The results of the user study (Section 9). Reviewers may still wish to use RESL to try the tasks given to user study participants, as detailed below.

Note: this VM runs the single-threaded version of the synthesizer rather than the multi-threaded version that ran on AWS. This should not affect the results of Section 8 as they do not include running times, but may cause using RESL through the Arena to fail to synthesize some more complex programs.

Section 8.1, Table 1

From directory ~Desktop/ run benchpred.

Script will output Table 1 in CSV format, excluding columns 2 (difficulty) and 3(completed online) which originate on each task's webpage. In column 4 (h(r)) in Table 1, height of programs is reported as zero-based (i.e., the height of a tree with only a literal is 0) whereas in our codebase the height is one-based (the height of a literal is 1). This means that 1 needs to be subtracted from the values in the table outputted by benchpred in this column to match the values in Table 1.

Section 8.2, Table 2

From directory ~Desktop/ run benchsketch.

Comparing these results to Table 2 is demonstrated with the following output:

new context size=2
|ext(E)|=9,i) => 10 + (10 * e))
outside equiv classes=412
inside equiv classes=412
discarded subtrees={}
all subtrees={10,e,10 * e}
  1. The first line is the benchmark name (column 1)
  2. The second line (new context size) is |V| (column 4)
  3. The third line (|E|) is column 2
  4. |ext(E)| is the column extend(E) (column 5)
  5. The fifth line is the target program. The portion of it being synthesized is inside the arrow function - in this example 10 + (10 * e). Column 4 of the table, terms(res), is derived from this subexpression, and counts the number of nodes in the AST, in this case 5 (10, ?+?, 10, ?*?, e). This also derives the structure of the tree in the last column.
  6. "outside equiv classes" is the number of equivalence classes seen when performing observational equivalence on the completed sketch (column 7).
  7. "inside equiv classes" is the number of equivalence classes seen when performing observational equivalence on the completion (column 6)
  8. The remaining rows are the means to drawing/reading the figure in the last column. "all subtrees" is the multiset of all expressions in the AST. Each of these is given a node. discarded subtrees shows every expression from this set that had an equivalent expression found before it in the enumeration on the full program. If there is no subexpression of this expression already discarded, the node will be colored black. The entire path to the root from a black node will be colored gray. This includes some expressions that appear in "discarded subtrees" whose subexpressions already appear in "discarded nodes". This is because in the test enumeration, the discarded nodes were not truly discarded, and therefore their superexpressions (which would not exist in a true OE on full program enumeraiton, and should be colored gray) may still end up being equivalent to another program and placed into the "discarded subtrees" set.

Evaluating RESL

Using the RESL Arena

You can use the RESL Arena (section 5 in the paper) to run synthesis queries. For example, you can try the tasks from the user study (section 9)

To run RESL, open an interactive shell and use the following commands: resl

You can try to solve the programming tasks we introduced in our user study:

Benchmark scripts

To run all benchmarks (~90 minutes): benchall

To run Predicate / Sketch benchmark only (~45 minutes each): benchpred benchsketch

Modifying the code:

The code for the synthesis engine is in the ~/Desktop/js-gentest directory. It is built with sbt. Below is a brief overview of the components.

To recompile the project's sources: rebuild

To run the project's unit tests after modifications: cd ~/Desktop/js-gentest; sbt test

RESL Components

RESL consists of 2 parts:

js-gentest: Synthesis logic, including benchmark calculation and synthesis server. The code resides in ~/Desktop/js-gentest/src/main/scala. Synthesis server's main function is in ~/Desktop/js-gentest/src/main/scala/Server/SynthesisServer.scala. Predicate benchmark's main function is in ~/Desktop/js-gentest/src/test/scala/Benchmark/PredicateBenchmarkDriver.scala. Sketch benchmark's main function is in ~/Desktop/js-gentest/src/test/scala/Benchmark/SketchBenchmarksDriver.scala.

resl-ui: Web interface for the RESL Arena. The code resides in ~/Desktop/resl-ui/{public,src}. Presented page's HTML is in ~/Desktop/resl-ui/public/index.html. RESL's functionality is in ~/Desktop/resl-ui/src/index.js.