This documents shows all performance results reported in the paper 
¨Quantitative Program Sketching using Lifted Static Analysis¨.
The reported results are obtained on the machine 
[11th Gen Intel(R) Core(TM) i7-1165G7 CPU@2.80GHz, 8 GB Memory, VM LUbuntu 20.10]

We compare three tools:
- FamilySketcher2 that uses the decision tree-based lifted analysis and the polyhedra domain from the APRON library.
- Program sketching tool Sketch version 1.7.6 that uses SAT-based inductive synthesis.
- The Brute Force enumeration approach that analyzes all variants, one by one, using a single-program analysis based on the polyhedra domain from the APRON library.


####################################################################################
## FamilySketcher2 --- Quantitative Program Sketcher using Lifted Static Analysis ## 
####################################################################################

FamilySketcher2 is a research prototype quantitative program sketcher designed for 
resolving numerical sketches using lifted static analysis based on abstract interpretation.


## Author

	Aleksandar Dimovski
	
	

# Examples 
```
cd family_sketcher2
```
All benchmarks from the paper are in ¨bench¨ subfolder of ¨family_sketcher2¨ folder. 
Enter the folder that contains the tool, and write the following commands. 
Note that after commands in square brackets we write the reported times in seconds on our machine 
[11th Gen Intel(R) Core(TM) i7-1165G7 CPU@2.80GHz, 8 GB Memory, VM LUbuntu 20.10], 
and in rounded brackets we write the obtained solutions. 

1. Loop1a example
Loop1a example with 5, 6, and 16 bits sizes of holes is given as loop1a-5.c, loop1a-6.c, and loop1a-16.c
$ ./Main.native -tree -domain polyhedra bench/loop1a-5.c	[0.016] (A=3 and B=0)
$ ./Main.native -tree -domain polyhedra bench/loop1a-6.c	[0.017] (A=3 and B=0)
$ ./Main.native -tree -domain polyhedra bench/loop1a-16.c	[0.017] (A=3 and B=0)

2. Loop1b example
Loop1b example with 5, 6, and 16 bits sizes of holes is given as loop1b-5.c, loop1b-6.c, and loop1b-16.c
$ ./Main.native -tree -domain polyhedra bench/loop1b-5.c   [0.026] (A=1 and B=0) (A=0 and B=0)
$ ./Main.native -tree -domain polyhedra bench/loop1b-6.c   [0.026] (A=1 and B=0) (A=0 and B=0)
$ ./Main.native -tree -domain polyhedra bench/loop1b-16.c  [0.027] (A=1 and B=0) (A=0 and B=0)

3. Loop2a example
Loop2a example with 5, 6, and 16 bits sizes of holes is given as loop2a-5.c, loop2a-6.c, and loop2a-16.c
$ ./Main.native -tree -domain polyhedra bench/loop2a-5.c	[0.060] (A=3 and B=0)
$ ./Main.native -tree -domain polyhedra bench/loop2a-6.c	[0.060] (A=6 and B=0)
$ ./Main.native -tree -domain polyhedra bench/loop2a-16.c	[0.061] (A=3 and B=0) 

4. Loop2b example
Loop1b example with 5, 6, and 16 bits sizes of holes is given as loop2b-5.c, loop2b-6.c, and loop2b-16.c
$ ./Main.native -tree -domain polyhedra bench/loop2b-5.c   [0.047] (A=1 and B=0) 
$ ./Main.native -tree -domain polyhedra bench/loop2b-6.c   [0.047] (A=1 and B=0)
$ ./Main.native -tree -domain polyhedra bench/loop2b-16.c  [0.049] (A=1 and B=0)

5. LoopCond example
LoopCond example with 5, 6, and 16 bits sizes of holes is given as loopcond-5.c, loopcond-6.c, and loopcond-16.c 
$ ./Main.native -tree -domain polyhedra bench/loopcond-5.c   [0.042] (A=2) 
$ ./Main.native -tree -domain polyhedra bench/loopcond-6.c   [0.042] (A=2)
$ ./Main.native -tree -domain polyhedra bench/loopcond-16.c  [0.043] (A=2)

6. NestedLoop example
NestedLoop example with 5, 8, and 16 bits sizes of holes is given as nestedloop-5.c, nestedloop-6.c, and nestedloop-16.c 
$ ./Main.native -tree -domain polyhedra bench/nestedloop-5.c   [0.126] (A=0 and B=0) 
$ ./Main.native -tree -domain polyhedra bench/nestedloop-6.c   [0.126] (A=0 and B=0) 
$ ./Main.native -tree -domain polyhedra bench/nestedloop-16.c  [0.128] (A=0 and B=0) 

6. Vmcai2004 example
NestedLoop example with 5, 8, and 16 bits sizes of holes is given as vmcai2004b-5.c, vmcai2004b-6.c, and vmcai2004b-16.c 
$ ./Main.native -tree -domain polyhedra bench/vmcai2004b-5.c   [4.69] (A=0 and B=30) (A=0 and B=31) 
$ ./Main.native -tree -domain polyhedra bench/vmcai2004b-6.c   [15.52] (A=0 and B=62) (A=0 and B=63) 
$ ./Main.native -tree -domain polyhedra bench/vmcai2004b-16.c  [TIMEOUT]




################################################################################
##Brute Force approach --- Single-Program Analysis of all variants one by one ##
################################################################################

We stay in the same folder ¨family_sketcher2¨ as for FamilySketcher2, we run the same example files, but now  
we use command-line option ¨-single¨ instead of ¨-tree¨. 

# Examples 
All benchmarks from the paper are in ¨bench¨ subfolder of ¨family_sketcher2¨ folder. 
Enter the folder that contains the tool, and write the following commands. 
Note that after commands in square brackets we write the reported times in seconds on our machine 
[11th Gen Intel(R) Core(TM) i7-1165G7 CPU@2.80GHz, 8 GB Memory, VM LUbuntu 20.10], 

1. Loop1a example
Loop1a example with 5, 6, and 16 bits sizes of holes is given as loop1a-5.c, loop1a-6.c, and loop1a-16.c
$ ./Main.native -single -domain polyhedra bench/loop1a-5.c		[4.66] 
$ ./Main.native -single -domain polyhedra bench/loop1a-6.c		[21.33] 
$ ./Main.native -single -domain polyhedra bench/loop1a-16.c		[TIMEOUT]    

2. Loop1b example
Loop1b example with 5, 6, and 16 bits sizes of holes is given as loop1b-5.c, loop1b-6.c, and loop1b-16.c
$ ./Main.native -single -domain polyhedra bench/loop1b-5.c   	[4.77] 
$ ./Main.native -single -domain polyhedra bench/loop1b-6.c   	[21.38] 
$ ./Main.native -single -domain polyhedra bench/loop1b-16.c  	[TIMEOUT] 

3. Loop2a example
Loop2a example with 5, 6, and 16 bits sizes of holes is given as loop2a-5.c, loop2a-6.c, and loop2a-16.c
$ ./Main.native -single -domain polyhedra bench/loop2a-5.c		[8.66] 
$ ./Main.native -single -domain polyhedra bench/loop2a-6.c		[42.81] 
$ ./Main.native -single -domain polyhedra bench/loop2a-16.c		[TIMEOUT]  

4. Loop2b example
Loop1b example with 5, 6, and 16 bits sizes of holes is given as loop2b-5.c, loop2b-6.c, and loop2b-16.c
$ ./Main.native -single -domain polyhedra bench/loop2b-5.c   	[8.45] 
$ ./Main.native -single -domain polyhedra bench/loop2b-6.c   	[36.04]
$ ./Main.native -single -domain polyhedra bench/loop2b-16.c  	[TIMEOUT] 

5. LoopCond example
LoopCond example with 5, 6, and 16 bits sizes of holes is given as loopcond-5.c, loopcond-6.c, and loopcond-16.c 
$ ./Main.native -single -domain polyhedra bench/loopcond-5.c   [1.19]  
$ ./Main.native -single -domain polyhedra bench/loopcond-6.c   [2.56] 
$ ./Main.native -single -domain polyhedra bench/loopcond-16.c  [TIMEOUT] 

6. NestedLoop example
NestedLoop example with 5, 8, and 16 bits sizes of holes is given as nestedloop-5.c, nestedloop-6.c, and nestedloop-16.c 
$ ./Main.native -single -domain polyhedra bench/nestedloop-5.c   [65.03] 
$ ./Main.native -single -domain polyhedra bench/nestedloop-6.c   [TIMEOUT] 
$ ./Main.native -single -domain polyhedra bench/nestedloop-16.c  [TIMEOUT] 

7. Vmcai2004 example
Vmcai2004 example with 5, 8, and 16 bits sizes of holes is given as vmcai2004b-5.c, vmcai2004b-6.c, and vmcai2004b-16.c 
$ ./Main.native -single -domain polyhedra bench/vmcai2004b-5.c   [5.12]  
$ ./Main.native -single -domain polyhedra bench/vmcai2004b-6.c   [19.12]  
$ ./Main.native -single -domain polyhedra bench/vmcai2004b-16.c  [TIMEOUT]



###########################################################################################
## Sketch 1.7.6 --- Program Sketcher available from https://people.csail.mit.edu/asolar/ ##
###########################################################################################

```
cd sketch-1.7.6
cd sketch-frontend
```

# from sketch-frontend directory you can test all sketches. 
Note that we use the following ¨./sketch¨ options

option --bnd-cbits determines the size in bits of control holes [default is 5]
option --bnd-inbits determines the size in bits of inputs [default is 5]
option --bnd-unroll-amnt determines the unroll amount for loops [default is 8]

# Examples from the paper are in ¨tests¨ folder: 
Note that after commands in square brackets we write the reported times in seconds on our machine 
[Intel Core TM i7-8700 CPU@3.20GHz x 12, 8 GB Memory, 64-bit Ubuntu 18.04.5 LTS]


1. Loop1a examples 
Loop1a example is given as loop1a.sk 
$ ./sketch --bnd-cbits 5 --bnd-inbits 5 --bnd-unroll-amnt 8 tests/loop1a.sk   [0.192]
$ ./sketch --bnd-cbits 6 --bnd-inbits 6 --bnd-unroll-amnt 8 tests/loop1a.sk   [0.197] 
$ ./sketch --bnd-cbits 16 --bnd-inbits 16 --bnd-unroll-amnt 8 tests/loop1a.sk   [37.74]

1. Loop1b examples 
Loop1b example is given as loop1b.sk
$ ./sketch --bnd-cbits 5 --bnd-inbits 5 --bnd-unroll-amnt 11 tests/loop1b.sk	[0.203]
$ ./sketch --bnd-cbits 6 --bnd-inbits 6 --bnd-unroll-amnt 11 tests/loop1b.sk	[0.216]
$ ./sketch --bnd-cbits 16 --bnd-inbits 16 --bnd-unroll-amnt 11 tests/loop1b.sk 	[2.44]

3. Loop2a examples 
Loop2a example is given as loop2a.sk
$ ./sketch --bnd-cbits 5 --bnd-inbits 5 --bnd-unroll-amnt 11 tests/loop2a.sk   	[0.200] [empty solution]
$ ./sketch --bnd-cbits 6 --bnd-inbits 6 --bnd-unroll-amnt 11 tests/loop2a.sk	[0.202] [empty solution]
$ ./sketch --bnd-cbits 16 --bnd-inbits 16 --bnd-unroll-amnt 11 tests/loop2a.sk	[0.348] [empty solution]

4. Loop2b examples 
Loop2b example is given as loop2b.sk
$ ./sketch --bnd-cbits 5 --bnd-inbits 5 --bnd-unroll-amnt 11 tests/loop2b.sk	[0.203] [empty solution]
$ ./sketch --bnd-cbits 6 --bnd-inbits 6 --bnd-unroll-amnt 11 tests/loop2b.sk	[0.205] [empty solution]
$ ./sketch --bnd-cbits 16 --bnd-inbits 16 --bnd-unroll-amnt 11 tests/loop2b.sk	[0.521] [empty solution] 

5. LoopCond examples 
LoopCond example is given as loopcond.sk 
$ ./sketch --bnd-cbits 5 --bnd-inbits 5 --bnd-unroll-amnt 32 tests/loopcond.sk			[0.207]
$ ./sketch --bnd-cbits 6 --bnd-inbits 6 --bnd-unroll-amnt 64 tests/loopcond.sk			[0.209] 
$ ./sketch --bnd-cbits 16 --bnd-inbits 16 --bnd-unroll-amnt 65536 tests/loopcond.sk   	[FAILS]

6. NestedLoop example
NestedLoop example is given as nestedloop.sk
$ ./sketch --bnd-cbits 5 --bnd-inbits 5 --bnd-unroll-amnt 8 tests/nestedloop.sk  	[FAILS]
$ ./sketch --bnd-cbits 6 --bnd-inbits 6 --bnd-unroll-amnt 8 tests/nestedloop.sk		[FAILS]
$ ./sketch --bnd-cbits 16 --bnd-inbits 16 --bnd-unroll-amnt 8 tests/nestedloop.sk	[FAILS]

7. Vmcai2004 example
Vmcai2004 example is given as vmcai2004.sk
$ ./sketch --bnd-cbits 5 --bnd-inbits 5 --bnd-unroll-amnt 8 tests/vmcai2004.sk		[0.192]
$ ./sketch --bnd-cbits 6 --bnd-inbits 6 --bnd-unroll-amnt 8 tests/vmcai2004.sk		[0.229]
$ ./sketch --bnd-cbits 16 --bnd-inbits 16 --bnd-unroll-amnt 8 tests/vmcai2004.sk	[0.292]