Report of Appliances Available for Testing - Deliverable D4.2

This deliverable reports on the two storage technologies from two vendors, DDN and Seagate, which were
designed to work with the Earth System Data Middleware (ESDM) - details of which were reported in
Deliverable D4.1 “Advanced software stack for Earth system data”[ESiWACE2 D4.1]. The expectation was
that various storage appliances would have support for ESDM and could be deployed with ESDM at the
weather and climate sites as ESDM would become available.
Seagate demonstrated the CORTX Motr (formerly Mero) software installation being capable of working
with commodity hardware, with the ability to be installed at the German Climate Computing Center (DKRZ)
and other centres in the future as per their requirements. This is a change in strategy from Seagate for
CORTX Motr. At the beginning of the project, the plan was to have an appliance based on Mero – as Mero
was closed source software. However, during the time frame of ESiWACE2, Mero was open sourced as
CORTX Motr – making it feasible to be deployed very widely on commodity hardware. Showcasing CORTX
Motr’s capability to work on commodity hardware working with ESDM, we believe is a powerful message
for the community rather than providing just a “closed” appliance. This deliverable hence takes that
approach.
DDN demonstrated the relevance and the efficiency of the Earth Systems Data Middleware (ESDM)
developed during the ESiWACE project on state-of-the-art industrial storage technologies. DDN
implemented a sophisticated integration of ESDM with the flash native, burst buffer solution product of
DDN’s portfolio, Infinite Memory Engine (IME). As part of the integration ESDM now supports the IME
native interface that bypasses the FUSE, allowing to obtain the maximum performance of IME. The code of
the integration is available and accessible to everyone through GitHub. Using IME, DDN was able to
produce performance results significantly beyond the state of the art in term of hardware / software
cooperation. Furthermore, ESDM provides performance in the range of the low-level API while still offering
a higher level of semantic to the end-user. The results were obtained within one of DDN’s labs using 5 IME
appliances and 10 clients. With an aggregated peak I/O bandwidth of 100 GB/sec, this setting is
representative of the core of the HPC / technical computing market. In addition to the lab set-up and the
integration effort, DDN has led more theoretical and research-oriented work. In respect to the Active
Storage task, DDN provided a complete design architecture and implemented a proof of concept (PoC). The
prototype is expected to be ready within the Q1 of 2023. The PoC code of the active storage is available on
GitHub.