Streaming Genomics at Scale: Efficient CSV Delivery via GraphQL, Brotli and WASM

PLEASE NOTE:  Following the conference, it is important to note that the approach described in this abstract was influenced by time and infrastructure constraints. While effective at the scale of ARGA’s datasets in this specific context, embedding large data transfers within a GraphQL response is not generally recommended. In most cases, a more conventional approach - such as generating an export job, writing the result to object storage (e.g., S3 or GCS), and providing download via HTTP - will be more scalable, maintainable, and operationally robust.

The Australian Reference Genome Atlas (ARGA), an advanced indexing platform designed to aggregate, filter, and provide access to complex genomics datasets, saw a need to support performant downloads of large datasets without introducing bespoke infrastructure. The primary objective was to enable users to efficiently retrieve large-scale CSV data through the existing GraphQL API. We also sought to avoid standalone download endpoints that would introduce maintenance overhead and architectural complexity. Our solution was two-pronged, leveraging backend and client-side implementations: 1. We leveraged the performance of ARGA s Rust-based backend and the flexibility of GraphQL. The backend retrieves the necessary records directly from ARGA s PostgreSQL database and transforms them into a CSV-formatted string. This CSV content is then compressed using Brotli, a modern compression algorithm known for its high compression ratio and decompression speed. The resulting binary data is encoded into a base64 string to ensure compatibility with the GraphQL transport layer and transmitted to the client through a standard GraphQL response. 2. On the client side, the base64 string is decoded, and WebAssembly (WASM) modules are used to decompress the Brotli-encoded data directly within the browser. This approach leverages WASM s efficiency and sandboxed execution to minimise latency and browser resource consumption. The solution achieved a significant reduction in payload size from 56.1 MB uncompressed down to 12.1 MB compressed for a sample dataset of 265,459 records. In this presentation, we demonstrate the novel utilisation of compression techniques to enhance the performance and efficiency of web-based applications.