Who is Afraid of a Petabyte Dataset? Rethinking Repository Infrastructures and Curation Workflows for the Scale and Type of Next Generation Data

As the US Department of Energy’s largest multi-program laboratory, Oak Ridge National Laboratory (ORNL) researchers are solving some of our world’s greatest challenges across a variety of domains. ORNL has world-leading facilities for neutron scattering, material and nanoscale research, additive manufacturing, high-performance computing, and several others. Embarking on a new initiative to connect instrument to compute, and with the advancement of machine learning (ML) and robotic automation, there are some key challenges for storing, sharing, and preserving these massive datasets. Constellation (https://doi.ccs.ornl.gov), our public data repository located at the Oak Ridge Leadership Facility (OLCF), is undergoing a redesign to address the next generation challenges of curating, sharing, and preserving petabyte files. In our presentation, we will discuss the challenges that we are facing and our proposed three-pronged approach: i) creatively repurposing existing tools and community development for automated curation; ii) identifying data “tiers” or stages based on preservation and discoverability needs; and iii) the creation of a domain-specific knowledge portal.