New Storage Layout for Earth System Data (D4.2)

Abstract

Making the best use of HPC in Earth simulation requires storing and manipulating vast quantities of data.

Existing storage environments face usability and performance challenges for both domain scientists and the data centers supporting the scientists.

These challenges arise from data discovery/access patterns, and the need to support complex legacy interfaces. In the ESiWACE project, we develop a novel I/O middleware targeting, but not limited to, earth system data.

This deliverable sheds light upon the technical design of the ESD middleware, and the user perspective and implications when using the middleware.

Its architecture builds on well-established end-user interfaces but utilizes scientific metadata to harness a data structure centric perspective.

In contrast to existing solutions, the middleware maps data structures to available storage technology based on several parameters:

1) A data center specific configuration of available hardware with their characteristics;

2) The intended usage pattern explicitly provided by the user and implicitly by the structure of the data.

This allows to exploit performance characteristics of a heterogeneous storage environment more efficiently.

This deliverable provides the background on data representations and description formats commonly used in earth system modeling.

The document isolates the key requirements for an earth system middleware and collects numerous use-case outlining the benefit to existing and anticipated workflows and technologies.

Finally, a detailed initial design for the architecture of the earth system middleware is proposed and documented.

The document is not intended to describe all components completely but provides a high-level overview that is necessary to build a first prototype as it is planned in the next phase of the ESiWACE project.

During this development, the design will be adjusted to match the prototype; the final version of the design document will be delivered with the end of the project.

About this document

Work package in charge: WP4 Exploitability

Actual delivery date for this deliverable: July 2017

Dissemination level: PU

Lead author: 

Deutsches Klimarechenzentrum GmbH (DKRZ), Jakob Lüttgau

Other contributing authors:

Deutsches Klimarechenzentrum GmbH (DKRZ), Julian Kunkel

Science and Technology Facilities Council (STFC), Bryan Lawrence, Jens Jensen

The University of Reading (UREAD), Bryan Lawrence

Seagate Systems UK Limited (SEAGATE), Giuseppe Congiu, Huang Hua

Fondazione Centro Euro-Mediterraneo sui Cambiamenti Climatici (CMCC): Paola Nassisi