Usage

ADCIRC+Dynamic Load Balancing (ADCIRC+DLB) operates in much the same way as default ADCIRC. However, the usage of the file pre-processor ADCPREP is now only used to spool NetCDF output files as the domain decomposition steps are taken care of transparently by ADCIRC+DLB at run time. The options used to control load balancing are specified in a simple ASCII text control file that must have the name LB.151, which is explained in detail below.

Execution

Prior to execution, the user executes ADCPREP, but with a new option 8 called prepareNetcdfFiles, which just populates the meta-data and the spools the desired NetCDF output files that were specified in the fort.15 file.

After running ADCPREP, the user runs the program as such:

mpirun -np N padcirc_dlb

where padcirc_dlb is the executable compiled with the dynamic load balancing capability and N represents the number of desired processors to utilize.

Control File - LB.151

If the LB.151 file is not located inside the directory, the program will default back to normal operation (i.e., static load balancing) and not deactivate/clip any dry elements.

The format of the LB.151 is detailed below. The file is space delimited and the variables that were used in the article have the same syntax as below.

0 # 1 (0) write (do not write) local subdomain files into `PEXXXX` directories.
1 # Number of user-defined load balancing events. Default is 1, which implies to start the simulation.
0 0.0005d0 -0.10d0 4 0 -0.10d0 0.10d0 # Each option is explained below in detail.

The 7 options for the 3rd line in the text block above are listed in the following order:

IT_restart, d_b, h_b, T_push, T_shrink, + Δ, − Δ

IT_restart: The integer timestep at which to force the adcirc+dlb program to rebalance the domain decomposition.
d_b: If a vertex's nearest distance from the wet/dry boundary exceeds a user-defined threshold d_b in WSGS84 degrees, the vertex is set to an offline-state. If the three vertices of an element all exceed d_b, then the element is set to an offline-state. The key assumption here is that locations in close proximity to the shoreline are most likely to be flooded.
h_b: A vertex's elevation must be located above a user-defined value h_b in meters above the model's geoid to be set to an offline-state. If the three vertices of an element all exceed h_b, then the element is set to an offline-state. A key assumption here is that areas that are low-lying are most likely to be flooded. In practice, the h_b can be selected based on the local inter-tidal range.

Warning

h_b is defined following adcirc convention so that above ground is negative and below the geoid (i.e., underwater) is positive.

T_push: If the time between two consectutive rebalance events is less than T_push, then h_b is increased by + Δ.
T_shrink: If the time between two consectutive rebalance events is greater than T_shrink, then h_b is decreased by − Δ.
+ Δ: Decimal percent to change the current h_b.
− Δ: Decimal percent to change the current h_b.

Note

Seperate values can be specified for + Δ and − Δ to give the user more control over lowering or raising the buffer.

Warning

Poor choices for the buffer options can lead to rebalancing events every simulation timestep, which will result in dramatic slow downs as pointed out in the article. Please consult the article [dlb] for the best performing options in a practical Hurricane simulation.

User-defined reblance events

The user can control when the model will rebalance by specifying an integer greater than 1 in the second line and then listing the 7 options for the buffer at that particular timestep they would like to enforce a rebalance event.

0 # 1 (0) write (do not write) local subdomain files into `PEXXXX` directories.
2 # Number of user-defined load balancing events (termed epochs).
0 0.0005d0 -0.10d0 4 0 -0.10d0 0.10d0 # Load balancing options for epoch 1
86400 0.0005d0 -0.50d0 4 0 -0.10d0 0.10d0 # Load balancing options for epoch 2

Otherwise, the program will automatically rebalance if the boundary of the buffer is wetted (i.e., checkpoint vertices). This behavior is explained in detail in the article [dlb].

Supported features

Dynamic load balancing fundamentally changes the behavior of the program and thus some features are not currently supported.

However, basic capaibilites were retained for basin-to-channel Hurricane simulation:

Meterological forcing from OceanWeatherIncorporated wind files (fort.22*)
All weakly enforced boundary conditions (IBTYPE > 20) including riverine discharge type boundary conditions.
Non-overlapped weir configurations (IBTYPE=23, IBTYPE=24)
Elevation specified boundary conditions such as tidal forcing.
Point and global elevation, depth-averaged velocity, and point and global meterological outputs; however, no harmonic analysis capabilities.

Warning

Currently ADCIRC+SWAN is not compatible with the DLB capability.

References

dlb(1,2): Roberts, K.J., D. Wirasaet, J.C. Dietrich, W.J. Pringle, J.J. Westerink. Dynamic Load Balancing for Predictions of Storm Surge and Coastal Flooding.