HiDALGO2 D2.4 Infrastructure Provisioning, Workflow Orchestration and Component Integration - revised

This deliverable provides an overview of the components, that make up the core foundation of the HiDALGO2 ecosystem and support the project’s use cases in achieving their scientific, technical and environmental goals. It builds upon the work of previous deliverables in HiDALGO2, as well as of D2.1, “Requirements Analysis and Scenario Definition”. It also collaborates with the development of D4.1, “Data Management and Coupling Technologies” and D3.1, “Scalability, Optimization and Co-Design Activities”. Here, HiDALGO2 outlines its technical infrastructure, consisting mainly of compute resources and web services, and lays out their distribution across the technical resources within the project, and the wider EuroHPC JU network. Individual components are broken down and described, and a map is developed of how they connect together to build up the high-level system context design. One of these components, the planned HiDALGO2 dashboard, is given a look ahead, with an outline of its design guidelines, and the foundations for its development in D2.7, a future deliverable. An in-depth description of the central components supporting the pilot use cases, the Workflow Orchestrators, MathSO and QCG, is given. A careful analysis of the general and specific requirements of the individual pilot simulations has been gathered from the partners, and a detailed development roadmap attempts to address these requirements, as well as introduce future features. The deliverable rounds out the system design of the HiDALGO2 environment with a strategy and vision for integrating all of its constituent components. The design of three distinct layers of component integration, or CI/CD pipelines, are shown in architecture diagrams following the C4 diagramming system, providing a successively lower level of detail in the system design. Finally, the roadmaps drawn out in the individual chapters give the direction the HiDALGO2 technical infrastructure and supporting development will follow, which will then be documented in future iterations of this deliverable, D2.6 and D2.7. In order to better address HiDALGO2’s strategy around continuous integration and continuous deployment (CI/CD) the chapter 5 has been rewritten. HiDALGO2’s efforts were highlighted in developing CI/CD for its services and its pilots, with the result of being able to deploy pilot use cases on multiple EuroHPC machines. It also emphasized the role that HiDALGO2 plays in collaboration and interoperability with CASTIEL2 to ensure communication with the planned public integration and deployment platform.