Presentation of paper "Prepare to Preserve: the Harvest Combine for Research Data at Stockholm University and the Systems Inventory"

ANATOMICALTHEATRE_20240917_1400_PHILIPSON_JOAKIM_V1

Presentation (with demo) of a short paper on the so-called Harvest Combine developed at Stockholm University, at iPRES2024, September 17. Abstract of paper follows:

In the absence yet of a digital archive compliant with the OAIS model requirements Stockholm University (SU) is nevertheless continually developing a "Harvest Combine" tool for harvesting and transformation of metadata and data files from data repositories that are used by SU researchers, such as Figshare, Datadryad and Zenodo. The metadata are collected from these repositories, enriched with metadata from other sources, and then transformed to accord with the Swedish National Archives recent implementation (aka FGS 2.0) of the European Common Specification for Information Packages (E-ARK CSIP) and Specification for Submission Information Packages (E-ARK SIP) versions 2.1.0. The metadata records are then stored in the SU local (temporal) archive together with the associated data files harvested simultaneously. Part of the motivation for this preparatory digital preservation work are the perceived risks of trusting the digital preservation of research data files produced by SU researchers to external repositories, that we do not fully control locally. The end product of this harvest and transformation processing are SIPs (Submission Information Packages), still awaiting future transformation to AIPs (Archival Information Packages) and eventually DIPs (Dissemination Information Packages). The associated data files are not transformed or converted in this first step towards long-term preservation and archiving, but we keep track of the file formats ingested partly by mapping file extensions to mime types in a special registry xml-file, used in the transformation processing and continuously updated whenever "new" file formats are encountered for the first time.

The software scripts that we developed for this "harvest combine" are in BASH Unix Shell, XQuery and XSLT. The processing of metadata, data and scripts occurs locally using Git Bash (for Windows), BaseX and Oxygen XML editor, but is essentially software-tool agnostic. Metadata input sources are e.g. OAI-PMH feeds (DataCite or METS), repository specific APIs (to get necessary file metadata) or generic command line scripts (e.g. for checksums).
Only recently we have also created a cross-sectional Digital Preservation Group at Stockholm University Library, whose first task is to create an inventory of external and internal source information systems (e.g. data repositories), together with their local destination archival or storage systems. This inventory is aimed to serve as a basis for a digital preservation plan for future long-term preservation.

Keywords – harvesting, metadata, transformation, research data, repositories, long-term preservation