HiDALGO2 D2.6 Infrastructure Provisioning, Workflow Orchestration and Component Integration

Deliverable D2.6 is the final report in a series of deliverables (D2.4, D2.5, D2.6) on HiDALGO2’s “Infrastructure Provisioning, Workflow Orchestration and Component Integration”. It describes the technical components and the infrastructure that make up HiDALGO2’s operational environment, providing a concise summary of the results achieved in the previous deliverables of the series, with focus kept on updates since the last reporting period, and on looking towards maintaining and improving the technical environment going into the final year of the HiDALGO2 project. 
HiDALGO2’s simulation and product development depend on different forms of compute infrastructure. Firstly, for faster development and prototyping, HiDALGO2 developers have access to in-house HPC, AI and HPDA resources. Secondly, Hi-DALGO2 has secured access to all available EuroHPC machines, for large scale runs of our simulation codes, and in the future, for scaling up AI and HPDA runs. HiDALGO2’s technical environment also consists of a considerable cloud resource allocation, to provision virtual machines to host the supporting and developmental services that HiDALGO2 development needs, as well as providing services for future external users and stakeholders. These services, and the resources behind them, as well as those around compute and HPDA, are reviewed shortly in chapter 2, while readers are referred to D2.4 and D2.5 for detailed descriptions.
Some of the main services provided through HiDALGO2, for both its developers and for external users, are the hybrid workflow orchestrators, namely, the MathSO Portal and the QCG Portal. These WFOs give the ability to deploy and run HiDALGO2 simulation codes on in-house as well as EuroHPC machines, along with features such as data orchestration and integration with visualisation tools. They are reviewed in depth in chapter 3, where an overview of the tools, as well as updates around their development is given, and detailed development roadmaps are provided. 
HiDALGO2 has also taken on the development of a suite of energy management tools for HPC applications. These are to be released directly on Jupiter, an exascale EuroHPC machine, and should provide benefits not only to HiDALGO2 pilot codes, but also the wider European HPC CoE network. Details and updates for its developments are given in chapter 4.
The last chapter is dedicated to the topic of component integration. D2.6 focuses on both the CI (continuous integration) and CD (continuous deployment) of the main products of HiDALGO2, the pilot simulation codes and their library frameworks, and how these codes and libraries can be efficiently and securely deployed to EuroHPC machines, through coordination and cooperation with CASTIEL2, for use by Hi-DALGO2 developers and future users, as well as any interested EuroHPC users.