Published October 8, 2022 | Version v1
Presentation Open

Running Ensemble Workflows at Extreme Scale: Lessons Learned and Path Forward

  • 1. Oak Ridge National Laboratory
  • 2. Lineberger Comprehensive Cancer Center, University of North Carolina, Chapel Hill, NC, USA
  • 3. Bredesen Center for Interdisciplinary Research and Graduate Education, University of Tennessee, Knoxville, TN, USA

Description

The ever-increasing volumes of scientific data com- bined with sophisticated techniques for extracting information from them have led to the increasing popularity of ensemble workflows which are a collection of runs of individual workflows. A traditional approach followed by scientists to run ensembles is to rely on simple scripts to execute different runs and manage resources. This approach is not scalable and is error-prone, thereby motivating the development of workflow management systems that specialize in executing ensembles on HPC clusters. However, when the size of both the ensemble and the target system reach extreme scales, existing workflow management systems face new challenges that hamper their efficient execution.

In this paper, we describe our experience scaling an ensemble workflow from the computational biology domain from the early design stages to the execution at extreme scale on Summit, a leadership class supercomputer at the Oak Ridge National Lab- oratory. We discuss challenges that arise when scaling ensembles to several million runs on thousands of HPC nodes. We identify challenges with composition of the ensemble itself, its execution at large scale, post-processing of the generated data, and scalability of the file system. Based on the experience acquired, we develop a generic vision of the capabilities and abstractions to add to existing workflow management systems to enable the execution of ensemble workflows at extreme scales. We believe that the un- derstanding of these fundamental challenges will help application teams along with workflow system developers with designing the next generation of infrastructure for composing and executing extreme-scale ensemble workflows.

Files

escience_2022.pdf

Files (1.9 MB)

Name Size Download all
md5:cadd956667da45539d262f29b03f13da
1.9 MB Preview Download