H5Intent: Autotuning HDF5 with user Intent

The complexity of data management in HPC systems stems from the diversity in I/O behavior exhibited by new workloads, multistage workflows, and multitiered storage systems. The HDF5 library is a popular interface to interact with storage systems in HPC workloads. The library manages the complexity of diverse I/O behaviors by providing user-level configurations to optimize the I/O for HPC workloads. The HDF5 library consists of hundreds of properties that provide various I/O optimizations and improve I/O performance for different use cases. However, these configurations are challenging to set by users who lack expertise in HDF5 library internals. We propose a paradigm change through our H5Intent software, where users specify the intent of I/O operations and the software can set various HDF5 properties automatically to optimize the I/O behavior. This work demonstrates several use cases that map user-defined intents to HDF5 configurations to optimize I/O. In this study, we make three observations. First, I/O intents can accurately define HDF5 configurations while managing conflicts and improving I/O performance by up to 22x. Second, I/O intents can be efficiently passed to HDF5 using the Intent Adaptor with a small footprint of 6.84 KB per node for 1000s intents per process. Third, H5Intent vol can dynamically map I/O intents to HDF5 configurations for various I/O behaviors exhibited by our microbenchmark and improve I/O performance by up to 8.8x. In conclusion, the H5Intent software optimizes complex large-scale HPC workloads such as VPIC and BD-CATS by up to 11x better I/O performance on the Lassen supercomputer.