Toward Cardiac Electrophysiology Digital Twins with an Efficient Open Source Scalable Solver on GPU Clusters

File description

The following repository contains all the necessary files to reproduce the manuscript "Toward Cardiac Electrophysiology Digital Twins with an Efficient Open Source Scalable Solver on GPU Clusters". Among the files are the following:

• monoalg3d_config_files.zip = All the configuration files for the computational simulations: "Benchmark myocardium cuboid", "Purkinje-Muscle-Junction calibration" and "Large-scale biventricular simulations";
• monoalg3d_purkinje_networks.zip = All the Purkinje networks and PMJ files for the "Purkinje-Muscle-Junction calibration" and "Large-scale biventricular simulations";
• monoalg3d_polaris_job_files.zip = All the job files to execute the "Benchmark myocardium cuboid", "Large-scale biventricular simulations" and "Purkinje-Muscle-Junction calibration" simulations on the Polaris supercomputer;
• monoalg3d_source_files.zip = the custom source files to build MonoAlg3D on Polaris. This file contains the implemented custom functions for the "domains_library", "extra_data_library" and "matrix_assembly_library". In addition, the modified version of the ToR-ORd cellular model for T-wave personalisation is also provided within the "models_library" folder;
• monoalg3d_ventricular_meshes.zip = All the ventricular meshes in ALG format used for the "Purkinje-Muscle-Junction calibration" and "Large-scale biventricular simulations".

Build and configuration

MonoAlg3D is developed by Rafael Sachetto Oliveira et al., Int J Numer Method Biomed Eng (2018) (the original solver is available on GitHub at: MonoAlg3D_C).

To configure the solver for this study follow the build procedures on the solver`s GitHub repository and copy the source files from monoalg3d_source_files.zip. Build the solver again with the command:

./build.sh -f

Biventricular meshes

The human biventricular mesh and ECG electrodes positions were derived from MRI data in Lyon et al., Front Physiol (2018) and Minchole et al., Front Physiol (2019) and can be found in https://zenodo.org/records/14046617 (referred to as Subject/Control 2 as in Camps et al., Med Image Anal (2024)). The ALG columns for the biventricular meshes have the following syntax:

54250,21250,250,250,250,250,0.000399119,1,0.249467,1,1,0.626517,-0.775963,-0.00453349,0.168679,0.130518,0.97453,0.758425,0.614972,-0.213462,7.15285928592859
54750,21250,250,250,250,250,0.0196224,1,0.245434,1,1,0.590492,-0.781949,0.0278572,0.136531,0.13876,0.979878,0.770388,0.574527,-0.192601,7.15285928592859
55250,21250,250,250,250,250,0.0190111,1,0.2414,1,1,0.587508,-0.783926,0.0437592,0.10975,0.136908,0.982621,0.776757,0.573408,-0.170214,7.15285928592859
55750,21250,250,250,250,250,0.0183998,1,0.237367,1,1,0.584524,-0.785903,0.0596612,0.0829692,0.135057,0.985363,0.783125,0.572289,-0.147828,7.3178097809781

• 1st Column: Coordinate x of the centre of the cell (units in micrometres);
• 2nd Column: Coordinate y of the centre of the cell (units in micrometres); 
• 3rd Column: Coordinate z of the centre of the cell (units in micrometres); 
• 4th Column: Half discretization in the x direction (units in micrometres); 
• 5th Column: Half discretization in the y direction (units in micrometres); 
• 6th Column: Half discretization in the z direction (units in micrometres); 
• 7th Column: Transmurality: range=[0, 1]; 
• 8th Column: Transmurality labels: 1-ENDO, 2-MID, 3-EPI;
• 9th Column: Base to apex heterogeneity: range=[0, 1]; 
• 10th Column: Fast endocardium layer: 0=FAST ENDO, 1=NORMAL, 2=DENSE, 3=SPARSE;
• 11th Column: Infarct stage: 1=Healthy, 2=Borderzone, 3=Infarcted-stage1, 4=Infarcted-stage2, ... 
• 12th-14th Columns: Fiber orientation f vector: fx, fy and fz values 
• 15th-17th Columns: Fiber orientation s vector: sx, sy and sz values
• 18th-20th Columns: Fiber orientation n vector: nx, ny and nz values
• 21th Column: IKs scaling factor. 

For the "Purkinje-Muscle-Junction calibration" simulations the dense/sparse biventricular wedges were generated using the "src/main_clip_mesh.c" script by extracting a part of the biventricular mesh with a box around the specified Purkinje terminals, as depicted on Figure 2A from the main manuscript.

Fibre fields were derived as in Doste et al., Int J Numer Method Biomed Eng (2019).

The fast endocardium layer region was generated following Camps et al., Med Image Anal (2024).

Scaling factors for the IKs current were calculated following Camps et al., Med Image Anal (2025).

Purkinje network

The Purkinje network and Purkinje-Muscle-Junctions were inferred in Camps et al., Med Image Anal (2024) by a minimal Purkinje network, which was later expanded by an extra branching procedure following Berg et al., Sci Rep (2023). 

Run and dispatch simulations

To execute the simulations located in the configuration files (monoalg3d_config_files.zip), just execute:

./bin/MonoAlg3D -c <configuration_file_path>

Simulations that use the MPI batch will need MPI configured and can be executed with:

mpiexec --np ${NMPIPROCESS} -ppn ${NRANKS_PER_NODE} --depth=${NDEPTH} --cpu-bind depth [GPU_AFFINITY_SCRIPT] ./bin/MonoAlg3D_batch -c <batch_configuration_file>

All simulations were dispatched on Polaris using the job files (monoalg3d_polaris_job_files.zip) with the command:

qsub <job_file>