Optimizing Workflow Performance by Elucidating Semantic Data Flow

The combination of ever-growing scientific datasets and the complexity of distributed workflows creates I/O performance bottlenecks due to data volume, velocity, and variety. While the increasing use of descriptive data formats (e.g., HDF5) helps organize these datasets, it also introduces obscure bottlenecks by requiring the translation of high-level operations into file addresses and subsequent low-level I/O operations. To address this challenge, we propose using Semantic Dataflow Graphs to analyze (a) relationships between logical datasets and file addresses, (b) how dataset operations translate into different I/O behaviors and their performance, and (c) the time-ordered relationship of tasks and data across entire workflows. Our analysis and visualization enable the identification of performance bottlenecks and reasoning about performance improvements in workflows.