Challenges in serving data using the HAPI interface

In the past, data management has traditionally embraced a setup based on the specific needs of each mission or instrument team.  Legacy systems developed applications that relied on data packaging on physical servers for on-premises traditional data centers to distribute.   One such system is the suite of tools, referred to as SDDAS, developed by Southwest Research Institute (SwRI).  SDDAS is built on a client/server model. The client side contains the analysis tools, catalog subsystem, and user interface. The catalog subsystem is particularly simple in that all entries are organized by time and are located using time as the independent variable.   The server side maintains the data archive and provides data granules (files) to the client side.  Because SDDAS allows data in distributed archives, data can be ordered and delivered over the Internet independent of the archive location and the nature of the archive storage.  One trade-off is that the data file(s) must be transferred to the client in their entirety, placing the burden on the client for retrieving the time interval requested.  This can result in a tremendous amount of data that must be transferred and maintained by the client. 

Although this data server model continues to serve its’ purpose, there is a recognition that publicly funded research and the underlying data should be open and easily accessible by larger communities of users.  The need for sharing forces a rethink of the way data delivery / availability is approached to make the data more easily accessible / mobile to wherever the use for the data may be.  The Heliophysics Application Programming Interface (HAPI) works towards this goal. The HAPI interface, a time series download and streaming format specification, provides a standard way to access Heliophysics data in order to free the user from having to know about file system boundaries, directory layouts, and file formats of the data sets of interest.  SwRI has incorporated the client-side software for HAPI in order to provide access to data that resides on various HAPI servers from within the SDDAS applications.  Since HAPI is not Heliophysics-specific, SwRI has also begun to use this standard as an add-on, not a replacement, for the ways in which SDDAS serves time series data.  As development begins, issues with archived data sets, such as instrument mode changes and data decimation, will begin to reveal how easy / hard it will be to become a HAPI compliant data server.