PIDs, Petabytes and Neutrons

Neutron science generates an exploding volume of scientific research data which needs to be managed under FAIR data principles. The European Spallation Source (ESS), is expected to generate tens of petabytes of data per year.

This volume of scientific research data benefits from a “PIDcentric” approach to manage the many science users, data sets and instruments.

In partnership with the Swiss and Swedish national synchrotron radiation facilities at PSI and MAXIV, ESS have developed SciCat, a new data catalogue. SciCat allows users to register and access data using persistent identifiers.

SciCat users are identified using the ORCID database, published datasets are identified using DOIs and  for raw and derived datasets, handle.net identifiers are used.

SciCat uses a document-oriented database, MongoDB, which allows datasets to be tagged with ad hoc, unstructured scientific metadata as well as traditionally structured metadata, which can then be recovered using the DOI or Handle.

In the day to day operations, SciCat also has to deal with the problem of legacy scientific research data. Legacy data with incomplete or lost metadata presents a challenge to include in data catalogue. In my session, I will talk about the ESS’s experiences with PIDs, data catalogues and legacy data.