Results of the 1st International Competition on Software Testing (Test-Comp 2019)

This file describes the contents of an archive of the
1st Competition on Software Testing (Test-Comp 2019)
https://test-comp.sosy-lab.org/2019/

The competition was run by Dirk Beyer, LMU Munich, Germany.
More information is available in the following article:
Dirk Beyer. First International Competition on Software Testing: Test-Comp 2019.
International Journal on Software Tools for Technology Transfer, 2020.

Copyright (C) Dirk Beyer
https://www.sosy-lab.org/people/beyer/

SPDX-License-Identifier: CC-BY-4.0
https://spdx.org/licenses/CC-BY-4.0.html

To browse the competition results with a web browser, there are two options:
- start a local web server using
  php -S localhost:8000
  in order to view the data in this archive, or
- browse https://test-comp.sosy-lab.org/2019/results/
  in order to view the data on the Test-Comp web page.

Contents:

index.html          directs to the overview web page
LICENSE.txt         specifies the license
README.txt          this file
results-validated/  results of validation runs
results-verified/   results of verification runs and aggregated results

The folder results-validated/ contains the results from validation runs:

- *.xml.bz2         XML results from BenchExec
- *.logfiles.zip    output from tools
- *.json.gz         mapping from files names to SHA 256 hashes for the file content

The folder results-verified/ contains the results from test-generation runs and aggregated results:

index.html               overview web page with rankings and score table
design.css               HTML style definitions
*.xml.bz2                XML results from BenchExec
*.merged.xml.bz2         XML results from BenchExec, status adjusted according to the validation results
*.logfiles.zip           output from tools
*.json.gz                mapping from files names to SHA 256 hashes for the file content
*.xml.bz2.table.html     HTML views on the detailed results data as generated by BenchExec's table generator
*.All.table.html         HTML views of the full benchmark set (all categories) for each tool
META_*.table.html        HTML views of the benchmark set for each meta category for each tool, and over all tools
<category>*.table.html   HTML views of the benchmark set for each category over all tools
iZeCa0gaey.html          HTML views per tool

quantilePlot-*           score-based quantile plots as visualization of the results
quantilePlotShow.gp      example Gnuplot script to generate a plot
score*                   accumulated score results in various formats

The hashes of the file names (in the files *.json.gz) are useful for
- validating the exact contents of a file and
- accessing the files from the witness store.

Overview over archives from Test-Comp 2019 that are available at Zenodo:

https://doi.org/10.5281/zenodo.3856669   Witness store (containing the generated test suites)
https://doi.org/10.5281/zenodo.3856661   Results (XML result files, log files, file mappings, HTML tables)
https://doi.org/10.5281/zenodo.3856478   Test tasks, version testcomp19
https://doi.org/10.5281/zenodo.2561835   BenchExec, version 1.18

All benchmarks were executed
for Test-Comp 2019, https://test-comp.sosy-lab.org/2019/
by Dirk Beyer, LMU Munich
based on the components
git@github.com:sosy-lab/sv-benchmarks.git  testcomp19-0-g6a770a9c1
git@gitlab.com:sosy-lab/test-comp/bench-defs.git  testcomp19-0-g1677027
git@github.com:sosy-lab/benchexec.git  1.18-0-gff72868

Feel free to contact me in case of questions:
https://www.sosy-lab.org/people/beyer/