PyTorrent: A Python Library Corpus for Large-scale Language Models

A large scale collection of both semantic and natural language resources is essential to leverage active Software Engineering research areas such as code reuse and code comprehensibility. Existing machine learning models ingest data from Open Source repositories (like GitHub projects) and forum discussions(like Stackoverflow.com), whereas, in this showcase, we took a step backward to orchestrate a corpus titled PyTorrent that contains 218,814 Python package libraries for the first time and collected from PyPI and Anaconda environment. This is because earlier studies have shown that much of the code is redundant and Python packages from these environments are better in quality and are well-documented. PyTorrent enables users (such as data scientists, students, etc.)to build off the shelf machine learning models directly without spending months of effort on large infrastructure

#Scenarios (datasets):

Please note there are two versions where V2 is slightly updated the content and compression with adding a Python tokenized code and revising the `path` with Python package name. The base schema is the same and more detail on latest schema can be found in PyTorrent GitHub Repository.

PyTorrent_Docstrings_v2.zip: Added Docstrings to the pairs of <NL,PL>

PyTorrent_UserComments_v2.zip: Added developer comments to the pairs of <NL,PL>

PyTorrent_Both_Docstrings_UserComments_v2.zip: Added both Docstrings and developer comments to the pairs of <NL,PL>

More detail can be found at PyTorrent GitHub Repository that includes metadata of each package, schema of metadata, dataset schema and how to use the dataset for training any machine-programming language based model. We also publish a pretrained model from PyTorrent and can be found at HuggingFace model Hub as PyTorrent-v1. 

Preprent: https://arxiv.org/pdf/2110.01710.pdf

Citation

@misc{bahrami2021pytorrent,
      title={PyTorrent: A Python Library Corpus for Large-scale Language Models}, 
      author={Mehdi Bahrami and N. C. Shrikanth and Shade Ruangwan and Lei Liu and Yuji Mizobuchi and Masahiro Fukuyori and Wei-Peng Chen and Kazuki Munakata and Tim Menzies},
      year={2021},
      eprint={2110.01710},
      archivePrefix={arXiv},
      primaryClass={cs.SE},
      howpublished={https://arxiv.org/abs/2110.01710},
}