Building the I (Interoperability) of FAIR for Performance Reproducibility of Large-Scale Composable Workflows in RECUP
Authors/Creators
- Nicolae, Bogdan (Researcher)1
-
Islam, Tanzima
(Researcher)2
-
Ross, Robert
(Researcher)1
-
Van Dam, Hubertus
(Researcher)3
-
Assogba, Kevin
(Researcher)4
-
Shpilker, Polina
(Researcher)5
-
Titov, Mikhail
(Researcher)3
-
Turilli, Matteo
(Researcher)3
- Wang, Tianle (Researcher)3
-
Jha, Shantenu
(Researcher)3
- et al. Show all 11 authors
- Nicolae, Bogdan (Researcher)1
-
Islam, Tanzima
(Researcher)2
-
Ross, Robert
(Researcher)1
-
Van Dam, Hubertus
(Researcher)3
-
Assogba, Kevin
(Researcher)4
-
Shpilker, Polina
(Researcher)5
-
Titov, Mikhail
(Researcher)3
-
Turilli, Matteo
(Researcher)3
- Wang, Tianle (Researcher)3
-
Jha, Shantenu
(Researcher)3
-
Pouchard, Line
(Researcher)3
-
1.
Argonne National Laboratory
-
2.
Texas State University
-
3.
Brookhaven National Laboratory
-
4.
Rochester Institute of Technology
-
5.
Tufts University
Description
Scientific computing communities increasingly run their experiments using complex data- and compute-intensive workflows that utilize distributed and heterogeneous architectures targeting numerical simulations and machine learning, often executed on the Department of Energy Leadership Computing Facilities (LCFs). We argue that a principled, systematic approach to implementing FAIR principles at scale, including fine-grained metadata extraction and organization, can help with the numerous challenges to performance reproducibility posed by such workflows. We extract workflow patterns, propose a set of tools to manage the entire life cycle of performance metadata, and aggregate them in an HPC-ready framework for reproducibility (RECUP). We describe the challenges in making these tools interoperable, preliminary work, and lessons learned from this experiment
Files
2023_escience-ReWords_workshop (1).pdf
Additional details
Dates
- Accepted
-
2023-09-13