Dataset Open Access

Natural Language-Guided Programming User Study

Heyman, Geert; Huysegems, Rafeal; Justen, Pascal; Van Cutsem, Tom


JSON-LD (schema.org) Export

{
  "description": "<p>In this dataset you find the&nbsp;user study data that was used in the <strong><em>Natural Language-Guided Programming</em></strong> paper, which is accepted for Onward! 2021. A preprint can be found here&nbsp;<a href=\"https://arxiv.org/pdf/2108.05198.pdf\">https://arxiv.org/pdf/2108.05198.pdf</a>. The dataset consists of the following files:</p>\n\n<ul>\n\t<li>\n\t<p>benchmark.json contains 201 test cases. Each test case consists of context, a natural language intent and target code. The test cases are intended to evaluate a model that can predict code giving a piece of context code and a natural language intent. The test cases were derived from Jupyter notebooks that were crawled from Github projects with permissive licenses. In the project_metadata field you find information about the original project such as its git url&nbsp;and&nbsp;license.</p>\n\t</li>\n\t<li>\n\t<p>predictions-annotated.json contains predictions of the three models used in the paper for 100 test cases in benchmark.json. Each prediction is accompanied with qualitive assesments from three annotators.</p>\n\t</li>\n\t<li>\n\t<p>train-index.jsonl is the list of github projects that were used for training the models.</p>\n\t</li>\n\t<li>\n\t<p>eval-index.jsonl is a list of github projects that we kept separate for evaluation. The benchmark.json was created from a random subset of the projects in this list.</p>\n\t</li>\n</ul>\n\n<p>For more details we refer to the paper.</p>", 
  "license": "https://opensource.org/licenses/BSD-3-Clause", 
  "creator": [
    {
      "affiliation": "Nokia Bell Labs", 
      "@id": "https://orcid.org/0000-0001-6276-424X", 
      "@type": "Person", 
      "name": "Heyman, Geert"
    }, 
    {
      "affiliation": "Nokia Bell Labs", 
      "@id": "https://orcid.org/0000-0001-6244-9864", 
      "@type": "Person", 
      "name": "Huysegems, Rafeal"
    }, 
    {
      "affiliation": "Nokia Bell Labs", 
      "@type": "Person", 
      "name": "Justen, Pascal"
    }, 
    {
      "affiliation": "Nokia Bell Labs", 
      "@id": "https://orcid.org/0000-0003-4116-4290", 
      "@type": "Person", 
      "name": "Van Cutsem, Tom"
    }
  ], 
  "url": "https://zenodo.org/record/5384768", 
  "datePublished": "2021-09-02", 
  "version": "0.0.1", 
  "keywords": [
    "code completion", 
    "code prediction", 
    "natural language-guided programming", 
    "example-centric programming"
  ], 
  "@context": "https://schema.org/", 
  "distribution": [
    {
      "contentUrl": "https://zenodo.org/api/files/30c22513-0bf3-4ee6-a71d-30ddc69eb860/benchmark.json", 
      "encodingFormat": "json", 
      "@type": "DataDownload"
    }, 
    {
      "contentUrl": "https://zenodo.org/api/files/30c22513-0bf3-4ee6-a71d-30ddc69eb860/eval-index.jsonl", 
      "encodingFormat": "jsonl", 
      "@type": "DataDownload"
    }, 
    {
      "contentUrl": "https://zenodo.org/api/files/30c22513-0bf3-4ee6-a71d-30ddc69eb860/predictions-annotated.json", 
      "encodingFormat": "json", 
      "@type": "DataDownload"
    }, 
    {
      "contentUrl": "https://zenodo.org/api/files/30c22513-0bf3-4ee6-a71d-30ddc69eb860/train-index.jsonl", 
      "encodingFormat": "jsonl", 
      "@type": "DataDownload"
    }
  ], 
  "identifier": "https://doi.org/10.5281/zenodo.5384768", 
  "@id": "https://doi.org/10.5281/zenodo.5384768", 
  "@type": "Dataset", 
  "name": "Natural Language-Guided Programming User Study"
}
66
4
views
downloads
All versions This version
Views 6666
Downloads 44
Data volume 16.9 MB16.9 MB
Unique views 5454
Unique downloads 11

Share

Cite as