Dataset Open Access

Natural Language-Guided Programming User Study

Heyman, Geert; Huysegems, Rafeal; Justen, Pascal; Van Cutsem, Tom


MARC21 XML Export

<?xml version='1.0' encoding='UTF-8'?>
<record xmlns="http://www.loc.gov/MARC21/slim">
  <leader>00000nmm##2200000uu#4500</leader>
  <datafield tag="653" ind1=" " ind2=" ">
    <subfield code="a">code completion</subfield>
  </datafield>
  <datafield tag="653" ind1=" " ind2=" ">
    <subfield code="a">code prediction</subfield>
  </datafield>
  <datafield tag="653" ind1=" " ind2=" ">
    <subfield code="a">natural language-guided programming</subfield>
  </datafield>
  <datafield tag="653" ind1=" " ind2=" ">
    <subfield code="a">example-centric programming</subfield>
  </datafield>
  <controlfield tag="005">20210903014840.0</controlfield>
  <controlfield tag="001">5384768</controlfield>
  <datafield tag="700" ind1=" " ind2=" ">
    <subfield code="u">Nokia Bell Labs</subfield>
    <subfield code="0">(orcid)0000-0001-6244-9864</subfield>
    <subfield code="a">Huysegems, Rafeal</subfield>
  </datafield>
  <datafield tag="700" ind1=" " ind2=" ">
    <subfield code="u">Nokia Bell Labs</subfield>
    <subfield code="a">Justen, Pascal</subfield>
  </datafield>
  <datafield tag="700" ind1=" " ind2=" ">
    <subfield code="u">Nokia Bell Labs</subfield>
    <subfield code="0">(orcid)0000-0003-4116-4290</subfield>
    <subfield code="a">Van Cutsem, Tom</subfield>
  </datafield>
  <datafield tag="856" ind1="4" ind2=" ">
    <subfield code="s">4169673</subfield>
    <subfield code="z">md5:e1080d051298444befa1014d4fa7bda0</subfield>
    <subfield code="u">https://zenodo.org/record/5384768/files/benchmark.json</subfield>
  </datafield>
  <datafield tag="856" ind1="4" ind2=" ">
    <subfield code="s">632582</subfield>
    <subfield code="z">md5:39f504c8d64f261b7862e3917aa259f7</subfield>
    <subfield code="u">https://zenodo.org/record/5384768/files/eval-index.jsonl</subfield>
  </datafield>
  <datafield tag="856" ind1="4" ind2=" ">
    <subfield code="s">6392629</subfield>
    <subfield code="z">md5:a50ad18a28bf8d1d750afba51118f00e</subfield>
    <subfield code="u">https://zenodo.org/record/5384768/files/predictions-annotated.json</subfield>
  </datafield>
  <datafield tag="856" ind1="4" ind2=" ">
    <subfield code="s">5692261</subfield>
    <subfield code="z">md5:cfd046716905235ee8f7139ee4ee240f</subfield>
    <subfield code="u">https://zenodo.org/record/5384768/files/train-index.jsonl</subfield>
  </datafield>
  <datafield tag="542" ind1=" " ind2=" ">
    <subfield code="l">open</subfield>
  </datafield>
  <datafield tag="260" ind1=" " ind2=" ">
    <subfield code="c">2021-09-02</subfield>
  </datafield>
  <datafield tag="909" ind1="C" ind2="O">
    <subfield code="p">openaire_data</subfield>
    <subfield code="o">oai:zenodo.org:5384768</subfield>
  </datafield>
  <datafield tag="100" ind1=" " ind2=" ">
    <subfield code="u">Nokia Bell Labs</subfield>
    <subfield code="0">(orcid)0000-0001-6276-424X</subfield>
    <subfield code="a">Heyman, Geert</subfield>
  </datafield>
  <datafield tag="245" ind1=" " ind2=" ">
    <subfield code="a">Natural Language-Guided Programming User Study</subfield>
  </datafield>
  <datafield tag="540" ind1=" " ind2=" ">
    <subfield code="u">https://opensource.org/licenses/BSD-3-Clause</subfield>
    <subfield code="a">BSD 3-Clause "New" or "Revised" License</subfield>
  </datafield>
  <datafield tag="650" ind1="1" ind2="7">
    <subfield code="a">cc-by</subfield>
    <subfield code="2">opendefinition.org</subfield>
  </datafield>
  <datafield tag="520" ind1=" " ind2=" ">
    <subfield code="a">&lt;p&gt;In this dataset you find the&amp;nbsp;user study data that was used in the &lt;strong&gt;&lt;em&gt;Natural Language-Guided Programming&lt;/em&gt;&lt;/strong&gt; paper, which is accepted for Onward! 2021. A preprint can be found here&amp;nbsp;&lt;a href="https://arxiv.org/pdf/2108.05198.pdf"&gt;https://arxiv.org/pdf/2108.05198.pdf&lt;/a&gt;. The dataset consists of the following files:&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;
	&lt;p&gt;benchmark.json contains 201 test cases. Each test case consists of context, a natural language intent and target code. The test cases are intended to evaluate a model that can predict code giving a piece of context code and a natural language intent. The test cases were derived from Jupyter notebooks that were crawled from Github projects with permissive licenses. In the project_metadata field you find information about the original project such as its git url&amp;nbsp;and&amp;nbsp;license.&lt;/p&gt;
	&lt;/li&gt;
	&lt;li&gt;
	&lt;p&gt;predictions-annotated.json contains predictions of the three models used in the paper for 100 test cases in benchmark.json. Each prediction is accompanied with qualitive assesments from three annotators.&lt;/p&gt;
	&lt;/li&gt;
	&lt;li&gt;
	&lt;p&gt;train-index.jsonl is the list of github projects that were used for training the models.&lt;/p&gt;
	&lt;/li&gt;
	&lt;li&gt;
	&lt;p&gt;eval-index.jsonl is a list of github projects that we kept separate for evaluation. The benchmark.json was created from a random subset of the projects in this list.&lt;/p&gt;
	&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For more details we refer to the paper.&lt;/p&gt;</subfield>
  </datafield>
  <datafield tag="773" ind1=" " ind2=" ">
    <subfield code="n">doi</subfield>
    <subfield code="i">isVersionOf</subfield>
    <subfield code="a">10.5281/zenodo.5384767</subfield>
  </datafield>
  <datafield tag="024" ind1=" " ind2=" ">
    <subfield code="a">10.5281/zenodo.5384768</subfield>
    <subfield code="2">doi</subfield>
  </datafield>
  <datafield tag="980" ind1=" " ind2=" ">
    <subfield code="a">dataset</subfield>
  </datafield>
</record>
66
4
views
downloads
All versions This version
Views 6666
Downloads 44
Data volume 16.9 MB16.9 MB
Unique views 5454
Unique downloads 11

Share

Cite as