There is a newer version of this record available.

Dataset Open Access

Complete Rxivist dataset of scraped bioRxiv data

Abdill, Richard J.; Blekhman, Ran

rxivist.org allows readers to sort and filter the tens of thousands of preprints posted to bioRxiv. Rxivist uses a custom web crawler to index all papers on biorxiv.org; this is a snapshot of Rxivist the production database. The version number indicates the date on which the snapshot was taken. See the included "README.md" file for instructions on how to use the "rxivist.backup" file to import data into a PostgreSQL database server.

Please note this is a different repository than the one used for the Rxivist manuscript—that is in a separate Zenodo repository. You're welcome (and encouraged!) to use this data in your research, but please cite our paper, now published in eLife.

Going forward, this information will also be available pre-loaded into Docker images, available at blekhmanlab/rxivist_data.

Version notes:

  • 2019-03-21
    • The license for this dataset has been changed to CC-BY, which allows use for any purpose and requires only attribution.
    • A new table, "publication_dates," has been added and will be continually updated. This table will include an entry for each preprint that has been published externally for which we can determine a date of publication, based on data from Crossref. (This table was previously included in the "paper" schema but was not updated after early December 2018.)
    • Foreign key constraints have been added to almost every table in the database. This should not impact any read behavior, but anyone writing to these tables will encounter constraints on existing fields that refer to other tables. Most frequently, this means the "article" field in a table will need to refer to an ID that actually exists in the "articles" table.
    • The "author_translations" table has been removed. This was used to redirect incoming requests for outdated author profile pages and was likely not of any functional use to others.
    • The "README.md" file has been renamed "1README.md" because Zenodo only displays a preview for the file that appears first in the list alphabetically.
    • The "article_ranks" and "article_ranks_working" tables have been removed as well; they were unused.
  • 2019-02-13.1
    • After consultation with bioRxiv, the "fulltext" table will not be included in further snapshots until (and if) concerns about licensing and copyright can be resolved.
    • The "docker-compose.yml" file was added, with corresponding instructions in the README to streamline deployment of a local copy of this database.
  • 2019-02-13
    • The redundant "paper" schema has been removed.
    • BioRxiv has begun making the full text of preprints available online. Beginning with this version, a new table ("fulltext") is available that contains the text of preprints that have been processed already. The format in which this information is stored may change in the future; any digression will be noted here.
    • This is the first version that has a corresponding Docker image.

Files (93.4 MB)
Name Size
1README.md
md5:3ebf2feedd85b09b8cd895610980e606
9.0 kB Download
docker-compose.yml
md5:2c16fd8916db8910b951af1b99503970
305 Bytes Download
rxivist.backup
md5:ac133a25568ba023b5c56289a1aa9f6b
93.4 MB Download
487
145
views
downloads
All versions This version
Views 48735
Downloads 14516
Data volume 7.0 GB467.0 MB
Unique views 36329
Unique downloads 929

Share

Cite as