Compact representation of the OpenAIRE citation graph

We're making available a distilled version of the OpenAIRE citation graph. We share the complete graph as two files totalling ~11GB. We also provide a larger file including additonal publication fields (see table below). For complete description see the preprint of the data paper at Arxiv.

There are three data files each served in two formats; the TSV (Tab-Separated Values) and Parquet :

• publications.tsv.xz and publications.parquet - The nodes in the citation graph, and their primary doi.
• citations.tsv.xz and citations.parquet - The edges in the citation graph
• publication_large.tsv.xz and publications_large.parquet - The nodes, but with several additional fields. 
• pipeline.tar.xz - The pyspark pipeline used to produce the other files. Contains a singularity/apptainer container for reproducability and portability. The code is also found at Codeberg.

All files are compressed (.xz or .parquet) files.

Memory efficient loading of the Parquet files using Pandas

For memory efficient loading of the citation.parquet and publications.parquet in loading, use the PyArrow backend:

import pandas as pd

df_pubs = pd.read_parquet(
    "publications.parquet",
    engine="pyarrow",
    dtype_backend="pyarrow",
)

df_cites = pd.read_parquet(
    "citations.parquet",
    engine="pyarrow",
    dtype_backend="pyarrow",
)

The large publications file e.g. publications_large.parquet, will not fit inside memory on most machines. It is however simple to select a subset of coloumns, and only load those. 

df_large = pd.read_parquet(
    "publications_large.parquet",
    columns=["nodeId", "title", "pid_dois"],
    engine="pyarrow",
    dtype_backend="pyarrow",
)

Here, the nodeId, title, and pid_dois columns are selected. 

The fields in the publication_large:

Memory usage is the size the column takes up in memory, when loaded into a panda using the code snippets below. The percentage of filled fields shows the amount of non-null entries in each column.