Revisiting MPI-IO Consistency: Potential Revisions and Their Impact on HDF5

The first Message Passing Interface (MPI) standard (MPI 1.0) was published in 1994, aiming to create a widely used standard for writing message-passing programs. However, I/O was not included in the initial MPI standard; support for parallel I/O was added later in MPI-2.0, which was published in 1997. Since then, I/O has become an integral part of the MPI standard. The design of MPI-IO has remained largely unchanged over the years, and we have observed some performance limitations in HPC systems.

MPI-IO serves as middleware that sits beneath the parallel HDF5 implementation. Therefore, HDF5's performance is significantly influenced by the underlying MPI-IO implementation. In this talk, I will discuss the performance implications of the current MPI-IO design, with a special focus on its interface and consistency semantics. Additionally, I will discuss an ongoing effort within the MPI-IO workgroup to revise the current MPI-IO consistency design, and how these potential revisions could affect HDF5, considering both programmability and performance.