eCSE-0302 Final Report

This report presents the work conducted for the ARCHER2 eCSE-0302 project. The project goal is to address the I/O bottleneck in the Xcompact3D CFD application by facilitating user defined in- situ analyses. This was achieved by first adding the ADIOS2 framework as an optional backend to the 2DECOMP&FFT library which Xcompact3D is built upon, providing run-time configurable I/O. The I/O configurations provided by ADIOS2 include redirecting the I/O to a concurrently running program instead of to disk. By also adding ADIOS2 to the Py4Incompact3D postprocessing tool, the data can be processed before writing to disk, limiting the volume of data written vs writing the entire 3-D solution field for later postprocessing which would additionally incur I/O costs to load the data. To be a practical solution project included work to parallelise the Py4Incompact3D tool. Testing disk- based I/O using the ADIOS2 backend showed it was capable of better scaling than was achieved by the current MPI-IO solution within 2DECOMP&FFT, which will be of benefit to other users of the 2DECOMP&FFT library. The Py4Incompact3D tool was successfully parallelised by wrapping the 2DECOMP&FFT library, demonstrating excellent scaling, allowing it to be used in larger cases than was previously practical. With the addition of ADIOS2 Py4Incompact3D was successfully coupled with a running Xcompact3D simulation on ARCHER2, performing the analysis in-situ. Scaling studies of the coupled Py4Incompact3D/Xcompact3D analysis showed performance comparable to Xcompact3D running in isolation until a 4:1 ratio of compute resources assigned to Xcompact3D over Py4Incompact3D was reached when load imbalance causes the coupled setup to become bottlenecked. This load balance crossover point was observed for a range of problem sizes and process counts up to a 5133 mesh size running on 4,096 Xcompact3D MPI ranks (for 1,024 Py4Incompact3D MPI ranks), beyond which the effects of load imbalance were observed.