Results of the 2nd International Competition on Software Testing (Test-Comp 2020)
Description
This archive contains the results of the
2nd Competition on Software Testing (Test-Comp 2020)
https://test-comp.sosy-lab.org/2020/
The competition was run by Dirk Beyer, LMU Munich, Germany.
More information is available in the following article:
Dirk Beyer. Second Competition on Software Testing: Test-Comp 2020.
In Proceedings of the 23rd International Conference on Fundamental Approaches to Software Engineering
(FASE 2020, Dublin, April 28-30), 2020. Springer.
https://doi.org/10.1007/978-3-030-45234-6_25
Copyright (C) Dirk Beyer
https://www.sosy-lab.org/people/beyer/
SPDX-License-Identifier: CC-BY-4.0
https://spdx.org/licenses/CC-BY-4.0.html
To browse the competition results with a web browser, there are two options:
- start a local web server using
php -S localhost:8000
in order to view the data in this archive, or
- browse https://test-comp.sosy-lab.org/2020/results/
in order to view the data on the Test-Comp web page.
Contents:
index.html directs to the overview web page
LICENSE.txt specifies the license
README.txt this file
results-validated/ results of coverage-validation runs
results-verified/ results of test-generation runs and aggregated results
The folder results-validated/ contains the results from coverage-validation runs:
- *.xml.bz2 XML results from BenchExec
- *.logfiles.zip output from tools
- *.json.gz mapping from files names to SHA 256 hashes for the file content
The folder results-verified/ contains the results from test-generation runs and aggregated results:
index.html overview web page with rankings and score table
design.css HTML style definitions
*.xml.bz2 XML results from BenchExec
*.merged.xml.bz2 XML results from BenchExec, status adjusted according to the validation results
*.logfiles.zip output from tools
*.json.gz mapping from files names to SHA 256 hashes for the file content
*.xml.bz2.table.html HTML views on the detailed results data as generated by BenchExec's table generator
*.All.table.html HTML views of the full benchmark set (all categories) for each tool
META_*.table.html HTML views of the benchmark set for each meta category for each tool, and over all tools
<category>*.table.html HTML views of the benchmark set for each category over all tools
iZeCa0gaey.html HTML views per tool
quantilePlot-* score-based quantile plots as visualization of the results
quantilePlotShow.gp example Gnuplot script to generate a plot
score* accumulated score results in various formats
The hashes of the file names (in the files *.json.gz) are useful for
- validating the exact contents of a file and
- accessing the files from the witness store.
Overview over archives from Test-Comp 2020 that are available at Zenodo:
https://doi.org/10.5281/zenodo.3678275 Witness store (containing the generated test suites)
https://doi.org/10.5281/zenodo.3678264 Results (XML result files, log files, file mappings, HTML tables)
https://doi.org/10.5281/zenodo.3678250 Test tasks, version testcomp20
https://doi.org/10.5281/zenodo.3574420 BenchExec, version 2.5.1
All benchmarks were executed
for Test-Comp 2020, https://test-comp.sosy-lab.org/2020/
by Dirk Beyer, LMU Munich
based on the components
git@github.com:sosy-lab/sv-benchmarks.git testcomp20-0-gd6cd3e5dd4
git@gitlab.com:sosy-lab/test-comp/bench-defs.git testcomp19-84-gac76836
git@github.com:sosy-lab/benchexec.git 2.5.1-0-gffad635
Feel free to contact me in case of questions:
https://www.sosy-lab.org/people/beyer/
Files
testcomp20-results.zip
Files
(983.5 MB)
Name | Size | Download all |
---|---|---|
md5:5c0c2e4b424071456ec04d6f75765b00
|
983.5 MB | Preview Download |