Dataset Open Access

Natural Language-Guided Programming User Study

Heyman, Geert; Huysegems, Rafeal; Justen, Pascal; Van Cutsem, Tom


Dublin Core Export

<?xml version='1.0' encoding='utf-8'?>
<oai_dc:dc xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/oai_dc/ http://www.openarchives.org/OAI/2.0/oai_dc.xsd">
  <dc:creator>Heyman, Geert</dc:creator>
  <dc:creator>Huysegems, Rafeal</dc:creator>
  <dc:creator>Justen, Pascal</dc:creator>
  <dc:creator>Van Cutsem, Tom</dc:creator>
  <dc:date>2021-09-02</dc:date>
  <dc:description>In this dataset you find the user study data that was used in the Natural Language-Guided Programming paper, which is accepted for Onward! 2021. A preprint can be found here https://arxiv.org/pdf/2108.05198.pdf. The dataset consists of the following files:


	
	benchmark.json contains 201 test cases. Each test case consists of context, a natural language intent and target code. The test cases are intended to evaluate a model that can predict code giving a piece of context code and a natural language intent. The test cases were derived from Jupyter notebooks that were crawled from Github projects with permissive licenses. In the project_metadata field you find information about the original project such as its git url and license.
	
	
	predictions-annotated.json contains predictions of the three models used in the paper for 100 test cases in benchmark.json. Each prediction is accompanied with qualitive assesments from three annotators.
	
	
	train-index.jsonl is the list of github projects that were used for training the models.
	
	
	eval-index.jsonl is a list of github projects that we kept separate for evaluation. The benchmark.json was created from a random subset of the projects in this list.
	


For more details we refer to the paper.</dc:description>
  <dc:identifier>https://zenodo.org/record/5384768</dc:identifier>
  <dc:identifier>10.5281/zenodo.5384768</dc:identifier>
  <dc:identifier>oai:zenodo.org:5384768</dc:identifier>
  <dc:relation>doi:10.5281/zenodo.5384767</dc:relation>
  <dc:rights>info:eu-repo/semantics/openAccess</dc:rights>
  <dc:rights>https://opensource.org/licenses/BSD-3-Clause</dc:rights>
  <dc:subject>code completion</dc:subject>
  <dc:subject>code prediction</dc:subject>
  <dc:subject>natural language-guided programming</dc:subject>
  <dc:subject>example-centric programming</dc:subject>
  <dc:title>Natural Language-Guided Programming User Study</dc:title>
  <dc:type>info:eu-repo/semantics/other</dc:type>
  <dc:type>dataset</dc:type>
</oai_dc:dc>
66
4
views
downloads
All versions This version
Views 6666
Downloads 44
Data volume 16.9 MB16.9 MB
Unique views 5454
Unique downloads 11

Share

Cite as