Forming Aggregations using Virtual Sharding: Lessons Learned from Simple Scalable Storage (S3)

Hyrax is a data server that allows scientific data access through DAP. It can access data from various sources, including Web Object Stores (WOS) like Amazon S3 and Google Cloud Storage (GCS) using Virtual Sharding.

Virtual Sharding is a technique that enables:

• Subsetting data directly from a WOS: Access specific portions of files/objects without downloading the entire file.
• Fast data aggregation: Combine data from multiple files/objects efficiently.

Benefits of Virtual Sharding:

• Faster data access: Subsetting and aggregation without moving the entire data.
• No data reformatting: Works directly with existing data structures like Cloud Optimized GeoTIFF and Zarr.

Previously, Hyrax required opening and processing each file for aggregation, which was slow. Virtual Sharding eliminates this step, significantly improving performance.

Overall, Hyrax with Virtual Sharding offers efficient data access and aggregation for scientific data stored using a WOS.