Published February 26, 2022
| Version v2
Dataset
Open
A Systematic Evaluation of Large Language Models of Code
Authors/Creators
- 1. Carnegie Mellon University
Description
These are datasets for the paper:
"A Systematic Evaluation of Large Language Models of Code"
https://arxiv.org/pdf/2202.13169.pdf
The code is available at: https://github.com/VHellendoorn/Code-LMs
The file "unseen_test_sets.tar.gz" contains test sets of ~100 files in each of 12 programming languages.
These files are not included in The Pile, and thus models such as GPT-Neo, GPT-J, GPT-NeoX were not trained on them.
In the paper, we use these test sets to compare a variety of language models of code including OpenAI's Codex, GPT-J, GPT-Neo, GPT-NeoX-20B, and CodeParrot and our PolyCoder model.
The file "index.zip" includes an index of the training set file paths and commit SHAs.
Notes
https://arxiv.org/abs/2202.13169