Dataset Open Access

Code4Bench: A Multidimensional Benchmark of Codeforces Data for Different Program Analysis Techniques

Majd Amirabbas; Vahidi-Asl Mojtaba; Khalilian Alireza; Baraani-Dastjerdi Ahmad; Zamani Bahman


DataCite XML Export

<?xml version='1.0' encoding='utf-8'?>
<resource xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://datacite.org/schema/kernel-4" xsi:schemaLocation="http://datacite.org/schema/kernel-4 http://schema.datacite.org/meta/kernel-4.1/metadata.xsd">
  <identifier identifierType="DOI">10.5281/zenodo.2582968</identifier>
  <creators>
    <creator>
      <creatorName>Majd Amirabbas</creatorName>
      <affiliation>Faculty of Computer Science and Engineering, Shahid Beheshti University G. C., Tehran, Iran</affiliation>
    </creator>
    <creator>
      <creatorName>Vahidi-Asl Mojtaba</creatorName>
      <affiliation>Faculty of Computer Science and Engineering, Shahid Beheshti University G. C., Tehran, Iran</affiliation>
    </creator>
    <creator>
      <creatorName>Khalilian Alireza</creatorName>
      <affiliation>Department of Software Engineering, Faculty of Computer Engineering, University of Isfahan, Isfahan, Iran</affiliation>
    </creator>
    <creator>
      <creatorName>Baraani-Dastjerdi Ahmad</creatorName>
      <affiliation>Department of Software Engineering, Faculty of Computer Engineering, University of Isfahan, Isfahan, Iran</affiliation>
    </creator>
    <creator>
      <creatorName>Zamani Bahman</creatorName>
      <affiliation>Department of Software Engineering, Faculty of Computer Engineering, University of Isfahan, Isfahan, Iran</affiliation>
    </creator>
  </creators>
  <titles>
    <title>Code4Bench: A Multidimensional Benchmark of Codeforces Data for Different Program Analysis Techniques</title>
  </titles>
  <publisher>Zenodo</publisher>
  <publicationYear>2019</publicationYear>
  <subjects>
    <subject>Reproduciblity</subject>
    <subject>Benchmark</subject>
    <subject>Software Testing</subject>
    <subject>Fault Localization</subject>
    <subject>Program Repair</subject>
  </subjects>
  <contributors>
    <contributor contributorType="Supervisor">
      <contributorName>Vahidi-Asl Mojtaba</contributorName>
      <affiliation>Faculty of Computer Science and Engineering, Shahid Beheshti University G. C., Tehran, Iran</affiliation>
    </contributor>
    <contributor contributorType="Supervisor">
      <contributorName>Haghighi Hasan</contributorName>
      <affiliation>Faculty of Computer Science and Engineering, Shahid Beheshti University G. C., Tehran, Iran</affiliation>
    </contributor>
  </contributors>
  <dates>
    <date dateType="Issued">2019-03-04</date>
  </dates>
  <language>en</language>
  <resourceType resourceTypeGeneral="Dataset"/>
  <alternateIdentifiers>
    <alternateIdentifier alternateIdentifierType="url">https://zenodo.org/record/2582968</alternateIdentifier>
  </alternateIdentifiers>
  <relatedIdentifiers>
    <relatedIdentifier relatedIdentifierType="DOI" relationType="IsVersionOf">10.5281/zenodo.2582967</relatedIdentifier>
  </relatedIdentifiers>
  <version>1.0.0</version>
  <rightsList>
    <rights rightsURI="http://creativecommons.org/licenses/by/4.0/legalcode">Creative Commons Attribution 4.0 International</rights>
    <rights rightsURI="info:eu-repo/semantics/openAccess">Open Access</rights>
  </rightsList>
  <descriptions>
    <description descriptionType="Abstract">&lt;p&gt;Reproducible research relies on well-designed benchmarks. However, evaluation on a single benchmark increases the risk of overfitting; that is, an optimization to reach a certain performance. In recent years several well-designed benchmarks have been constructed for different subfields of program analysis. However, they often involve real-world industrial projects in few languages such as C or Java. We provide Code4Bench, a benchmark comprising 3,421,357 programs totaling of 306,053,105 lines of code in 41 versions of 28 programming languages such as C/C++, Java, Python, and Kotlin. We have constructed this benchmark from Codeforces, a famous programming competition website, which is widely used by international programmers. Code4Bench advances the state-of-the-art in conducting reproducible and comparative experiments. It helps mitigate the bias and increase the generality and conclusiveness of the results. We present our methodology in construction of Code4Bench and give various descriptive statistics. We have also conducted an online survey on the users of Codeforces&amp;rsquo; website whose code is included in the benchmark. The survey is concerned about the user&amp;rsquo;s demographic information and programming habits, whose results are also provided in the benchmark. Finally, we leveraged an automatic process by which we localized faults within the faulty versions and categorize them according to a coarse-grained classification. In addition to its usage in empirical studies, Code4Bench can be used to teach programming and evolve algorithmic problems. We release Code4Bench in database format to allow researchers to extract other data of the benchmark by arbitrary queries.&lt;/p&gt;

&lt;p&gt;Code4Bench version 1.0.0 is publicly available at &lt;a href="https://zenodo.org/record/2582968"&gt;https://zenodo.org/record/2582968&lt;/a&gt;, with DOI 10.5281/zenodo.2582968, thereby providing long-term storage and versioning. It is released under the terms of Creative Commons Attribution 4.0 International license. Code4Bench is also publicly available at: &lt;a href="https://github.com/code4bench/Code4Bench"&gt;https://github.com/code4bench/Code4Bench&lt;/a&gt;, in which we have provided some additional information and script examples.&lt;/p&gt;</description>
  </descriptions>
</resource>
539
79
views
downloads
All versions This version
Views 539539
Downloads 7979
Data volume 50.2 GB50.2 GB
Unique views 494494
Unique downloads 5050

Share

Cite as