Semantic storage of climate data on object stores

New methods for storing large volumes of data, such as cloud systems and object storage present opportunities: • Improved access and user management over POSIX • Parallel reads / writes • Read / write from anywhere (no need to mount disks) and challenges: • Interface and access methods for users are changed • Exploit parallel and remote access nature of stores with S3 HTTP API • While maintaining a familiar and consistent API for the user • S3-netCDF-python: • Library to read/write netCDF 3 and netCDF 4 via a S3 HTTP API • Files can be streamed directly to memory or cached to disk • Files can be split into smaller objects • Multiple objects can be streamed in parallel