Towards a sustainable astronomical data infrastructure: Optimising linking data from the Rucio datalake to the users areas within the SKA Regional Centres Network

The distributed architecture of the SKA Regional Centre Network (SRCNet) aims to provide scientific communities worldwide with efficient computational and storage resources to exploit the massive data volumes produced by the SKA Observatory (SKAO). Given the amount of SKAO data, traditional data management paradigms — where data is transferred to computational resources— are no longer feasible. Instead, computational workflows must increasingly be relocated closer to data storage locations, emphasizing efficient data access strategies and avoiding unnecessary duplication or redundancy. In this context, we present PrepareData, a modular and extensible data delivery service developed within SRCNet prototyping activities. Our proposal for this service addresses the critical challenge of redundant data transfers and duplication at both node and user levels by enabling seamless delivery of requested datasets from local Rucio Storage Elements (RSEs) directly into users' working environments. PrepareData operates as a local service within each SRCNet node and it is integrated into a broader ecosystem of federated services. Specifically, we designed and evaluated two distinct yet complementary implementations to avoid unnecessary data duplication and to enable a dynamic data bridge between the RSEs and the user storage areas, through: (1) a filesystem-based solution leveraging CephFS, which uses shared filesystem mount points and bind mounts to ensure consistent and immediate data availability of the data across computational nodes, and (2) a Kubernetes model using Persistent Volumes and Persistent Volume Claims, dynamically injecting data into a user's areas. To tackle this work we detail the architectural design and development, the technical implementation, the integration of both solutions with science enabling tools, such as JupyterHub, CARTA or virtually any application, and finally we provide a performance evaluation. This contribution provides a scalable and sustainable blueprint for data delivery in federated scientific infrastructures, supporting the broader goals of green computing and efficient resource utilisation.

Modern scientific projects such as the Square Kilometre Array Observatory (SKAO) produce extremely large amounts of data — far more than traditional research systems have handled before. Scientists around the world need efficient ways to access and work with these vast datasets without unnecessary delays or wasteful copying of information. In the SKA Regional Centre Network (SRCNet), data are stored in distributed storage systems. However, these storage systems are not directly visible to the software tools scientists use for analysis, such as interactive notebooks, visualisation platforms, or specialised processing services. To make the data usable, it must be moved or linked into the scientists' working environments. Our research focuses on a service called PrepareData, designed to manage this delivery process in a way that is both fast and resource-efficient. We describe and test different technical methods for exposing data to users, including techniques that avoid repeatedly copying large files. We ran experiments to measure how fast and scalable each method is under realistic conditions. The results show that by linking data instead of copying it, PrepareData can reduce delays and lower the burden on storage systems. This leads to better performance for scientists and less wasted computing and storage resources. These improvements are especially important for major international science projects, where efficient data delivery can accelerate research and reduce environmental impact.