Poster Open Access
Nobody needs yet an other data format for HPC. But why have so-called self-describing data formats never provided out-of-the-box cross application portability? Why are most open-access datasets not self-describing for both the domain scientist and after-use? And why do communities need to implement their data readers in various post-processing, visualization and analysis frameworks over and over again?
We present the open meta data format openPMD for data format agnostic markup of particle-mesh data. Based on a minimal kernel of meta information and enriched with domain-specific extensions, we develop an open ecosystem of interoperable simulations and data processing frameworks from the domains of laser-plasma interaction, X-ray photon sciences, astrophysics up to systems biology. This poster presents our efforts to enable & establish workflows suitable to frictionless transposition between those domains, using highly scalable I/O methods (e.g. ADIOS BP or HDF5), a truly self-describing data markup and peer reviewed participation.
A. Huebl et al. On the Scalability of Data Reduction Techniques in Current and Upcoming HPC Systems from an Application Perspective, ISC 2017, arXiv:1706.00522
A. Huebl et al. openPMD 1.0.0: A meta data standard for particle and mesh based data, technical specification (CC-BY 4.0), November 2015, DOI:10.5281/zenodo.33624
C. Docan et al. DataSpaces: An interaction and coordination framework or coupled simulation workflows, HPDC 2010, DOI:10.1007/s10586-011-0162-y
H. Abbasi et al. Datastager: scalable data staging services for petascale applications, Cluster Computing 13(3), DOI:10.1007/s10586-010-0135-6