OPENDAP/bes: BES-3.21.0-46 for Hyrax-1.17.0

News for 3.21.0-46

Reworked the way we implement the BES's singletons (or more of them at least) so that they use the Meyers Singleton pattern. This includes the EffectiveUrlCache and TheBESKeys classes.

We have made significant advances to supporting HDF5 files in the DMR++ builder and interpreter. Our support is close to complete with only a very few features not supported. This support now includes HDF5's Varying-length string arrays. The software now also supports scalar strings in all their forms. Note that we also added support for fixed length string arrays in the DMR++ builder and interpreter.

This version of the BES includes a new feature, dependent on the newest version of libdap, that reduces the time needed to build NetCDF4 responses by a factor of 10 or more when the variables in the response are not spatially subset (that is, when all variables in the response are returned in their entirety). It is still possible to subset at the variable level and get this performance boost. This is the 'Direct I/O' feature added ti libdap. NOTE: To use this feature, you must rebuild DMR++ files using the version of the DMR++ builder associated with this version of the BES (or get_dmrpp tool from the hyrax-1.16.8-356 build).

We improved the performance of finding the effective URL for a data item when it is accessed via a series of redirect operations, the last of which is a signed AWS URL. This is a common case for data stored in S3.

We have added generic Memory and File caching, tailored specifically toward the cases that arise when serving data from S3 using the DMR++ system.

The BES now has much better support for response size and time limits. Users are warned about responses that are too large before they are built and the BES now exits gracefully when a response takes too long to build. The limits are configurable.

We replaced a home-grown HTTP connection pool with a scheme provided by libcurl. The two have equivalent performance, but the libcurl version is much easier to maintain and might offer room for improvement in the future.

C99 compatibility improvements for the GCTP code in/used-by the HDF5 handler. Thanks ti Florian Weimer for those fixes.

We have added a BES module that can work with S3 using the DMR++ system. This provides a data flow that is similar to the one we provide for Hyrax in the Cloud as developer for NASA, but this new module does not make use of the NASA/ESDIS CMR system to resolve 'NASA Granules' to URLs. This will enable other groups to use the DMR++ system to serve data from S3.

We have adopted C++-14 as aggressively as we can, resulting in fewer lines of code, better memory management and more efficient code.

We have moved more of the handlers to build/use DAP4 as the default response format and build DAP2 responses from those. This reduces the amount of code we have to maintain.

The BES can sign S3 URLs using the AWS V4 signing scheme. This uses the Credentials Manager system.

The bundled grid() and geogrid() server side functions now support DAP4.