Scalability and profiling of high-scale environmental simulations in cutting-edge HPC systems: status and prospects

Simulations have become a prevalent method in the scientific community for researching climate, environmental, and social phenomena. These simulations aid in understanding how various elements such as air, pollution, smoke, and heat disperse in complex spatial environments. Computational Fluid Dynamics (CFD) is a widely used application for conducting such simulations. It is a rapidly advancing field within computational sciences that enables for the creation of flow simulations based on the governing laws of fluid movement. By utilizing specialized data structures and numerical analysis, problems related to fluid flows can be effectively solved. These calculations simulate the flow of both liquid and gas, as well as their interactions with surfaces defined by boundary conditions. OpenFOAM is a leading CFD application that utilizes the finite volume method (FVM) to solve partial differential equations (PDE), and it is an open-source framework.

Due to the significant use of computational resources by simulation models, it is necessary to thoroughly understand the direction of development of the technologies on which modern HPC systems are based, which in turn is the key to building efficient applications that take full advantage of the offered hardware capabilities.

The primary motivation for preparing this work is to assess scalability and emerging application and hardware limitations by better considerate the available possibilities. In particular, the analysis is based on HPC resources, where we strive for trade-offs and correlations between the application and the hardware. Overall goal is to reduce performance bottlenecks and overcome memory limitations to improve overall performance. To achieve these goals, we use a variety of profiling scenarios and deep code analysis.

Simulation applications developed in the HiDALGO2 project that are used in environmental and social analyses were used as a reference point. We can distinguish among them: Urban Air Pollution (simulating air flow and pollution dispersion in an urban area), Urban Building Model (building models for effective integration with urban architecture), Renewable Energy Sources (advance energy production estimation from wind farms and solar panels) and Wildfires (simulate wildfire-atmosphere interactions and smoke dispersion).

These applications use various simulation frameworks for calculations (e.g. OpenFOAM, EULAG, Ktirio), which makes it possible to evaluate the performance of a wide range of applications in the context of HPC computing systems offered by EuroHPC Joint Undertaking. As a result of the conducted research, several potential bottlenecks related to I/O operations were identified, which will serve as the first target of optimization activities in the next period of the project implementation.

Moreover, the work presents the systematic and repeatable methodology used to collect and store benchmark results for all HiDALGO2 pilot projects to enable their development and optimization towards achieving the highest possible performance when running on EuroHPC JU supercomputers.

Furthermore, the study presents the possibilities of using innovative HPC technologies offered by AMD processors, which are the basis for the construction of modern HPC clusters on which the presented research was conducted (EuroHPC JU systems). The benchmark suite has been defined for in-depth profiling of AMD EPYC 9004 series processors (Genoa, Genoa X, Bergamo). Particular emphasis is placed on the impact of large-cache systems on the parallel performance of various OpenFOAM computational kernels. Many tests showed an optimal balance between memory speed and core count for a Genoa-based system with 2x64 cores, which is especially noticeable with smaller mesh sizes.