gprMax + MPI: Large-scale open-source computational electrodynamics

gprMax is an open-source software package for computational electrodynamics. Designed with a general electromagnetic solver engine, it has a set of core features developed for modelling Ground Penetrating Radar (GPR) and can be used to model electromagnetic wave propagation in fields such as engineering, geophysics, archaeology, and medicine. gprMax is a Python codebase with performance-critical parts using Cython.

This poster showcases the impact of research software engineers on enabling scientific research. Before this project, gprMax simulations could run using a CPU solver parallelised using OpenMP or a GPU solver developed using the NVIDIA CUDA programming model. However, these solvers ran on a single CPU node or GPU, preventing the full utilisation of HPC systems and limiting models to those that fit in the memory of a single node. Our work dramatically increases the modelling capabilities of gprMax by adding a Message Passing Interface (MPI) domain decomposition to the CPU solver that divides models between multiple nodes/MPI tasks. This can reduce the time to results when running on a single node, but more importantly, enables researchers to scale the size and/or resolution of their electromagnetic simulations, allowing them to utilise HPC systems and explore new science.

Additional improvements of interest to the RSE community include a Python interface to support file formats for parallel I/O (VTKHDF), a new suite of regression tests built for HPC using the ReFrame framework, and work to create visualisations of the model geometry and resultant electromagnetic field to demonstrate the new functionality in an accessible way.

This work was funded under the embedded CSE programme of the ARCHER2 UK National Supercomputing Service (https://www.archer2.ac.uk).