Dataset Open Access

Code4Bench: A Multidimensional Benchmark of Codeforces Data for Different Program Analysis Techniques

Majd Amirabbas; Vahidi-Asl Mojtaba; Khalilian Alireza; Baraani-Dastjerdi Ahmad; Zamani Bahman


JSON-LD (schema.org) Export

{
  "inLanguage": {
    "alternateName": "eng", 
    "@type": "Language", 
    "name": "English"
  }, 
  "description": "<p>Reproducible research relies on well-designed benchmarks. However, evaluation on a single benchmark increases the risk of overfitting; that is, an optimization to reach a certain performance. In recent years several well-designed benchmarks have been constructed for different subfields of program analysis. However, they often involve real-world industrial projects in few languages such as C or Java. We provide Code4Bench, a benchmark comprising 3,421,357 programs totaling of 306,053,105 lines of code in 41 versions of 28 programming languages such as C/C++, Java, Python, and Kotlin. We have constructed this benchmark from Codeforces, a famous programming competition website, which is widely used by international programmers. Code4Bench advances the state-of-the-art in conducting reproducible and comparative experiments. It helps mitigate the bias and increase the generality and conclusiveness of the results. We present our methodology in construction of Code4Bench and give various descriptive statistics. We have also conducted an online survey on the users of Codeforces&rsquo; website whose code is included in the benchmark. The survey is concerned about the user&rsquo;s demographic information and programming habits, whose results are also provided in the benchmark. Finally, we leveraged an automatic process by which we localized faults within the faulty versions and categorize them according to a coarse-grained classification. In addition to its usage in empirical studies, Code4Bench can be used to teach programming and evolve algorithmic problems. We release Code4Bench in database format to allow researchers to extract other data of the benchmark by arbitrary queries.</p>\n\n<p>Code4Bench version 1.0.0 is publicly available at <a href=\"https://zenodo.org/record/2582968\">https://zenodo.org/record/2582968</a>, with DOI 10.5281/zenodo.2582968, thereby providing long-term storage and versioning. It is released under the terms of Creative Commons Attribution 4.0 International license. Code4Bench is also publicly available at: <a href=\"https://github.com/code4bench/Code4Bench\">https://github.com/code4bench/Code4Bench</a>, in which we have provided some additional information and script examples.</p>", 
  "license": "http://creativecommons.org/licenses/by/4.0/legalcode", 
  "creator": [
    {
      "affiliation": "Faculty of Computer Science and Engineering, Shahid Beheshti University G. C., Tehran, Iran", 
      "@type": "Person", 
      "name": "Majd Amirabbas"
    }, 
    {
      "affiliation": "Faculty of Computer Science and Engineering, Shahid Beheshti University G. C., Tehran, Iran", 
      "@type": "Person", 
      "name": "Vahidi-Asl Mojtaba"
    }, 
    {
      "affiliation": "Department of Software Engineering, Faculty of Computer Engineering, University of Isfahan, Isfahan, Iran", 
      "@type": "Person", 
      "name": "Khalilian Alireza"
    }, 
    {
      "affiliation": "Department of Software Engineering, Faculty of Computer Engineering, University of Isfahan, Isfahan, Iran", 
      "@type": "Person", 
      "name": "Baraani-Dastjerdi Ahmad"
    }, 
    {
      "affiliation": "Department of Software Engineering, Faculty of Computer Engineering, University of Isfahan, Isfahan, Iran", 
      "@type": "Person", 
      "name": "Zamani Bahman"
    }
  ], 
  "url": "https://zenodo.org/record/2582968", 
  "datePublished": "2019-03-04", 
  "version": "1.0.0", 
  "keywords": [
    "Reproduciblity", 
    "Benchmark", 
    "Software Testing", 
    "Fault Localization", 
    "Program Repair"
  ], 
  "@context": "https://schema.org/", 
  "distribution": [
    {
      "contentUrl": "https://zenodo.org/api/files/413af3d7-1f54-430c-8840-33e53ade7e46/code4bench.rar", 
      "@type": "DataDownload", 
      "fileFormat": "rar"
    }
  ], 
  "identifier": "https://doi.org/10.5281/zenodo.2582968", 
  "@id": "https://doi.org/10.5281/zenodo.2582968", 
  "@type": "Dataset", 
  "name": "Code4Bench: A Multidimensional Benchmark of Codeforces Data for Different Program Analysis Techniques"
}
482
69
views
downloads
All versions This version
Views 482482
Downloads 6969
Data volume 43.9 GB43.9 GB
Unique views 441441
Unique downloads 4242

Share

Cite as