unarXive: All arXiv Publications Pre-Processed for NLP, Including Structured Full-Text and Citation Network (open subset)

Description

unarXive is a scholarly data set containing publications' structured full-text, annotated in-text citations, linked non-text content (mathematical notation, figure/table captions) and a citation network.

The data is generated from all LaTeX sources on arXiv and therefore of higher quality than data generated from PDF files.

Typical uses are

• Training of ML models (citation recommendation, summarization, LLMs)
• Citation context analysis
• Bibliographic analyses

Access

┏━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃  D O W N L O A D   S A M P L E   ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━┛

Regarding the full data set, please note the following:

Note: this Zenodo record is the "open subset" of unarXive, which contains all permissively licensed papers from arXiv.org. You can find the full version here.

The code used for generating the data set is publicly available.