Planned intervention: On Wednesday April 3rd 05:30 UTC Zenodo will be unavailable for up to 2-10 minutes to perform a storage cluster upgrade.
Published August 6, 2019 | Version v1
Report Open

Consistency and Collaboration for Fine-Grained Scientific Workflow Development: The dispel4py Information Registry

  • 1. NCSR "Demokritos"
  • 2. The University of Edinburgh

Description

This paper reports experience designing technology to support large-scale distributed computations that arise in scientific research as well as in other modern contexts.  The challenge is to support data-intensive work across multiple autonomous sites, where experimentation and collaborative development are simultaneously encouraged across the same computing infrastructure. Focusing on fine-grained streaming workflows for data-intensive tasks, and in particular on requirements arising through the use of dispel4py within the eScience context, we specify appropriate registry modules and their interactions with other core components, designed to achieve the aforementioned goals. We then discuss the design and usage of a prototype information registry designed to support Dispel and dispel4py workflows. Finally, we demonstrate our method's suitability through a seismic ambient-noise cross-correlation example, drawn from the field of seismology.

 

Files

dispel4py_registry_2015.pdf

Files (677.9 kB)

Name Size Download all
md5:756a358fb39dfb15a462eac0652e0074
677.9 kB Preview Download

Additional details

Funding

VERCE – Virtual Earthquake and seismology Research Community in Europe e-science environment 283543
European Commission