Linking provenance with system logs: a context aware information integration and exploration framework for analyzing workflow execution
Description
When executing scientific workflows in a distributed
environment, anomalies of the workflow behavior are often
caused by a mixture of different issues, e.g., careless design
of the workflow logic, buggy workflow components, unexpected
performance bottlenecks or resource failure at the underlying
infrastructure. The provenance information only defines data
evolution at the workflow level, which does not have an explicit
connection with the system logs provided by the underlying
infrastructure. Analyzing provenance information and apposite
system metrics requires expertise and a considerable amount of
manual effort. Moreover, it is often time-consuming to aggregate
this information and correlate events occurring at different levels
in the infrastructure. In this paper, we propose an architecture
to automate the integration among the workflow provenance
information with the performance information collected from
infrastructure nodes running workflow tasks. Our architecture
enables workflow developers or domain scientists to effectively
browse workflow execution information together with the system
metrics, and analyze contextual information for possible anomalies.
Files
2019.conference.workshop.iwsg-cam.pdf
Files
(633.3 kB)
Name | Size | Download all |
---|---|---|
md5:20ab8e0bfc5f7dd1bd094fe5fa1b4991
|
633.3 kB | Preview Download |
Additional details
Funding
- VRE4EIC – A Europe-wide Interoperable Virtual Research Environment to Empower Multidisciplinary Research Communities and Accelerate Innovation and Collaboration 676247
- European Commission
- ENVRI PLUS – Environmental Research Infrastructures Providing Shared Solutions for Science and Society 654182
- European Commission
- SWITCH – Software Workbench for Interactive, Time Critical and Highly self-adaptive cloud applications 643963
- European Commission
- ARTICONF – smART socIal media eCOsytstem in a blockchaiN Federated environment 825134
- European Commission
- ENVRI-FAIR – ENVironmental Research Infrastructures building Fair services Accessible for society, Innovation and Research 824068
- European Commission