Consistency and Collaboration for Fine-Grained Scientific Workflow Development: The dispel4py Information Registry
- 1. NCSR "Demokritos"
- 2. The University of Edinburgh
Description
This paper reports experience designing technology to support large-scale distributed computations that arise in scientific research as well as in other modern contexts. The challenge is to support data-intensive work across multiple autonomous sites, where experimentation and collaborative development are simultaneously encouraged across the same computing infrastructure. Focusing on fine-grained streaming workflows for data-intensive tasks, and in particular on requirements arising through the use of dispel4py within the eScience context, we specify appropriate registry modules and their interactions with other core components, designed to achieve the aforementioned goals. We then discuss the design and usage of a prototype information registry designed to support Dispel and dispel4py workflows. Finally, we demonstrate our method's suitability through a seismic ambient-noise cross-correlation example, drawn from the field of seismology.
Files
dispel4py_registry_2015.pdf
Files
(677.9 kB)
Name | Size | Download all |
---|---|---|
md5:756a358fb39dfb15a462eac0652e0074
|
677.9 kB | Preview Download |