Uncharted Territory - Exploring New Frontiers for HDF5 (Keynote)

Blast-off: GPU Accelerated HDF5
The industry's shift to accelerated computing has moved the goal posts for accessing application data, and provided dramatically faster compute resources for HDF5 transforms. This talk describes a path to directly accessing GPU memory for data access operations and moving HDF5 data transforms from the host to the GPU.

We're Breaking Up! Disaggregated HDF5 Containers on Object Storage Systems
Object storage systems dominate cloud computing and are making significant inroads into the on-premise storage deployments as well. This talk describes storing HDF5 containers (files, today) natively on object storage systems in high-performance and portable way. Disaggregating HDF5 metadata from application data and storing it in a lightweight database, e.g. SQLite, provides improvements to performance and resiliency, and adds query capabilities that are expensive with today's native file format. Sharding application data into multiple objects enables greater parallelism and performance as well, especially with cloud object stores.

Not for Spotify: Streaming HDF5 Data
HDF5 is ubiquitous across science domains, but has typically targeted array-like access patterns. Streaming data from cameras, particle accelerators, and other high-speed data sources into HDF5 files is also common. This talk presents new HDF5 APIs and format extensions that are tailored to this use case and maximize performance for both writing and reading data in a streamed fashion.