Planned intervention: On Wednesday April 3rd 05:30 UTC Zenodo will be unavailable for up to 2-10 minutes to perform a storage cluster upgrade.
Published October 31, 2019 | Version v1
Conference paper Open

Workflow environments for advanced cyberinfrastructure platforms

Description

Progress in science is deeply bound to the effective use of high-performance computing infrastructures and to the efficient extraction of knowledge from vast amounts of data. Such data comes from different sources that follow a cycle composed of pre-processing steps for data curation and preparation for subsequent computing steps, and later analysis and analytics steps applied to the results. However, scientific workflows are currently fragmented in multiple components, with different processes for computing and data management, and with gaps in the viewpoints of the user profiles involved. Our vision is that future workflow environments and tools for the development of scientific workflows should follow a holistic approach, where both data and computing are integrated in a single flow built on simple, high-level interfaces. The topics of research that we propose involve novel ways to express the workflows that integrate the different data and compute processes, dynamic runtimes to support the execution of the workflows in complex and heterogeneous computing infrastructures in an efficient way, both in terms of performance and energy. These infrastructures include highly distributed resources, from sensors and instruments, and devices in the edge, to High-Performance Computing and Cloud computing resources. This paper presents our vision to develop these workflow environments and also the steps we are currently following to achieve it.

Files

ICDCS_submission.pdf

Files (1.1 MB)

Name Size Download all
md5:e845d9fe91bbf8208d8cc86e980bb70e
1.1 MB Preview Download

Additional details

Funding

mF2C – Towards an Open, Secure, Decentralized and Coordinated Fog-to-Cloud Management Ecosystem 730929
European Commission
CLASS – Edge and CLoud Computation: A Highly Distributed Software Architecture for Big Data AnalyticS 780622
European Commission