This artifact contains the exported docker container that can be imported to reproduce the results shown in the paper submitted. A more detailed overview of using this is present in the attached README. 

To start simply import the tar in the following way:

`docker import apislicer-3.0.tar.gz <new_image_name>`

Following that you can start a container from the image using the following command:

docker run -it <new_image_name> /bin/bash

Now you're inside the docker container and can follow the instructions below to reproduce the results.

APISlicer Artifact

The docker image provided includes all the libraries and clients on which the evaluation was conducted in addition to the necessary scripts in order to reproduce the results.

The libraries used for the evaluation have all been installed with coverage enabled. You can find all the library source files under /tmp/benchmarks. All the clients of the libraries are located under /tmp/clients. For each library we created a folder which includes the manually cleaned test cases and the a directory which includes all the scripts necessary to reproduce the generation of the automatically generated test cases. Since the manually constructed test cases can not be automatically reproduced they are provided as is.

Generating Test cases using APISlicer

We are going to demonstrate how to generate the test cases for one client for one library but the same exact steps can be followed for any other client for any library. In our demonstration we will use xmlsec for libxml2.

Primitive client function selection

• Browse to /tmp/clients/libxml2_tests/xmlsec.
• In order to generate the test cases using function selection on client functions with primitive types you can use the script xmlsec.sh.
• Make sure that the test_cases_mode1 directory is empty before performing the run in order to avoid file name collisions or overwriting generated test cases.
• As the script is running you will see it invoking APISlicer on each source file in the xmlsec source directory with the corresponding includes extracted from compile_commands.json of xmlsec for that file. This run can take a few minutes depending on how large the client is.

root@77a2d77be6f2:/tmp/clients/libxml2_tests/xmlsec# ./xmlsec.sh
Executing:  ['/tmp/clients/libxml2_tests/xmlsec/run_apislicer.sh', '/tmp/clients/xmlsec1-1.2.37/apps/cmdline.c', '/usr/local/include/libxml2', '--includes=/tmp/clients/xmlsec1-1.2.37/apps,/tmp/clients/xmlsec1-1.2.37,/tmp/clients/xmlsec1-1.2.37/include,/tmp/clients/xmlsec1-1.2.37/include,/usr/local/include/libxml2,/tmp/benchmarks/libxml2/include']
Executing:  ['/tmp/clients/libxml2_tests/xmlsec/run_apislicer.sh', '/tmp/clients/xmlsec1-1.2.37/apps/crypto.c', '/usr/local/include/libxml2', '--includes=/tmp/clients/xmlsec1-1.2.37/apps,/tmp/clients/xmlsec1-1.2.37,/tmp/clients/xmlsec1-1.2.37/include,/tmp/clients/xmlsec1-1.2.37/include,/usr/local/include/libxml2,/tmp/benchmarks/libxml2/include']
Executing:  ['/tmp/clients/libxml2_tests/xmlsec/run_apislicer.sh', '/tmp/clients/xmlsec1-1.2.37/apps/xmlsec.c', '/usr/local/include/libxml2', '--includes=/tmp/clients/xmlsec1-1.2.37/apps,/tmp/clients/xmlsec1-1.2.37,/tmp/clients/xmlsec1-1.2.37/include,/tmp/clients/xmlsec1-1.2.37/include,/usr/local/include/libxml2,/tmp/benchmarks/libxml2/include']

• When APISlicer has run on all the source files you can see the autogenerated source files under test_cases_mode1.
• Running ls ./test_cases_mode1 | wc -l will give you the total functions that were selected in this mode. This number includes the cases where our generated failed to compile and these files are marked with a suffix "failed".

Primitive API function selection

• In the same folder you should find another script xmlsec_enhanced.sh which will run APISlicer in the second mode.
• Make sure that the test_cases_mode2 directory is empty before performing the run in order to avoid file name collisions or overwriting generated test cases.
• By running that script you should see run_apislicer_enhanced.sh being invoked on each source file of the client, in this case xmlsec.

root@77a2d77be6f2:/tmp/clients/libxml2_tests/xmlsec# ./xmlsec_enhanced.sh
Executing:  ['/tmp/clients/libxml2_tests/xmlsec/run_apislicer_enhanced.sh', '/tmp/clients/xmlsec1-1.2.37/apps/cmdline.c', '/usr/local/include/libxml2', '--includes=/tmp/clients/xmlsec1-1.2.37/apps,/tmp/clients/xmlsec1-1.2.37,/tmp/clients/xmlsec1-1.2.37/include,/tmp/clients/xmlsec1-1.2.37/include,/usr/local/include/libxml2,/tmp/benchmarks/libxml2/include']
Executing:  ['/tmp/clients/libxml2_tests/xmlsec/run_apislicer_enhanced.sh', '/tmp/clients/xmlsec1-1.2.37/apps/crypto.c', '/usr/local/include/libxml2', '--includes=/tmp/clients/xmlsec1-1.2.37/apps,/tmp/clients/xmlsec1-1.2.37,/tmp/clients/xmlsec1-1.2.37/include,/tmp/clients/xmlsec1-1.2.37/include,/usr/local/include/libxml2,/tmp/benchmarks/libxml2/include']
Executing:  ['/tmp/clients/libxml2_tests/xmlsec/run_apislicer_enhanced.sh', '/tmp/clients/xmlsec1-1.2.37/apps/xmlsec.c', '/usr/local/include/libxml2', '--includes=/tmp/clients/xmlsec1-1.2.37/apps,/tmp/clients/xmlsec1-1.2.37,/tmp/clients/xmlsec1-1.2.37/include,/tmp/clients/xmlsec1-1.2.37/include,/usr/local/include/libxml2,/tmp/benchmarks/libxml2/include'

• After a few minutes you can see the the auto-generated test cases in test_cases_mode2.
• Similar to the previous mode running ls ./test_cases_mode2 | wc -l will give you the total functions that were selected in this enhanced mode. As before, this number includes the cases where our generated failed to compile and these files are marked with a suffix "failed".

The directory where the cases are generated are in run_apislicer.sh and run_apislicer_enhanced.sh. It's possible to edit those two scripts to generate the test cases in different directories if required.

Measuring coverage

Each library is already built and installed with coverage enabled. In order to get the baseline coverage for the library we need first to run the regression test suite of the library. We are going to use libsodium to show how to measure the baseline coverage and how to measure coverage of the manually cleaned up tests.

• Browse to /tmp/benchmarks/libsodium-stable where the source directory of libsodium is.
• Run reset_cov.sh in order to delete any *.gcda files present in the current directory to make sure that the baseline measurement is accurate.
• Run make check which will invoke the regression test suite of the library.
• Once the tests have all been run this will create the *.gcda files which we can use to measure the baseline coverage.
• Run coverage_measurement.sh <dir_name> which will run lcov and genhtml to measure the coverage of the library after running the regression test suite. For example, you can run coverage_measurement.sh baseline which will create a directory baseline_out which has all the results of the coverage. Note that the coverage measure excludes test directories which you can verify in the coverage_measurement.sh script itself.
• You can download this directory to view it on a browser on a local machine by using the docker cp command. There will be an index.html file inside the directory which when opened using any browser will display the results. The result of running the coverage_measurement.sh script should display the following

Writing directory view page.
Overall coverage rate:
  lines......: 86.5% (8840 of 10221 lines)
  functions..: 91.3% (861 of 943 functions)
  branches...: 67.1% (1554 of 2317 branches)

• After getting the baseline coverage now it's time to run each test case in the manually cleaned test cases to see the difference in coverage.
• Browse to the folder where the manually cleaned test cases are, in this case it would be under /tmp/clients/libsodium_tests/cleaned_tests.
• In that folder there will be a compile_case.sh file which can be used to compile each test case and run it. The process would be to run compile_case.sh <file_name> and then run a.out to execute the test case. For example, to compile and execute one of the libsodium manually cleaned test cases you can run compile_case.sh key_derivation_test_get_key_from_password.c this will produce a file a.out in the current directory which you can execute. Follow, the same steps for all files in the cleaned_tests directory. Note that some test cases might take longer than others to execute.
• Now browse back to the libsodium source directory and run the coverage_measurement.sh script again. This time you can run the script with a different directory name. For example, you can run coverage_measurement.sh withtests which will create a folder with the coverage results. In our case after executing the three test cases you can see the coverage for libsodium being the following:

Writing directory view page.
Overall coverage rate:
  lines......: 87.6% (8952 of 10221 lines)
  functions..: 91.9% (867 of 943 functions)
  branches...: 69.1% (1601 of 2317 branches)

Each library has the a reset_cov.sh and coverage_measurement.sh script in it's source directory to measure coverage. The coverage_measurement.sh script includes the necessary exclusions from each directory to measure coverage only for the source code for the library. Each folder in /tmp/clients/lib* has a cleaned_cases folder which includes the selected test cases and the compilation script used to compile them. The same process can be followed for all the other libraries.