D5.3 Tools for integrating dynamic and real-time data streams

This deliverable presents the initial results of Task 5.3, Provision of Interfaces for Dynamic and Real-Time Data, within the LUMI AI Factory Service Center. The report outlines the technical, architectural, and operational requirements for integrating dynamic and real-time data streams into the LUMI environment, and assesses the current capabilities of the infrastructure in relation to these needs. Dynamic data refers to continuously changing information generated by sensors, user interactions, or automated systems. Its integration into high-performance computing (HPC) environments is essential for enabling responsive, data-driven AI workflows. The report identifies key challenges such as low-latency ingestion, protocol interoperability, edge computing, and secure data handling. The first version of the LUMI AI Factory tools includes components for high-speed data transfer, stream processing, edge integration, and observability. These tools are designed to support heterogeneous use cases, including environmental monitoring, predictive maintenance, and smart infrastructure. A comparative analysis highlights areas where LUMI’s current capabilities align with requirements and where further development is needed. Case studies, such as the SeaBee coastal monitoring pilot, demonstrate practical implementation scenarios and inform future development priorities. The report concludes with a roadmap for enhancing real-time data support, including ingestion middleware, protocol translation services, and AI/ML inference capabilities. It is likely that streaming data use cases are so heterogeneous that there cannot be one solution architecture. The end-users need to adapt the solution of their use cases. LUMI AI Factory can provide open source software component in containers and provide guidance for them.