DDS: integrating data analytics transformations intask-based workflows

High-performance data analytics (HPDA) is a current

trend in e-science research that aims to integrate traditional

HPC with recent data analytic frameworks. Most of the work

done in this field has focused on improving data analytic frameworks

by implementing their engines on top of HPC technologies

such as Message Passing Interface. However, there is a lack

of integration from an application development perspective.

HPC workflows have their own parallel programming models,

while data analytic (DA) algorithms are mainly implemented

using data transformations and executed with frameworks like

Spark. Task-based programming models (TBPMs) are a very

efficient approach for implementing HPC workflows. Data analytic

transformations can also be decomposed as a set of

tasks and implemented with a task-based programming model.

In this paper, we present a methodology to develop HPDA

applications on top of TBPMs that allow developers to combine

HPC workflows and data analytic transformations seamlessly.

A prototype of this approach has been implemented on top of

the PyCOMPSs task-based programming model to validate two

aspects: HPDA applications can be seamlessly developed and

have better performance than Spark. We compare our results

using different programs. Finally, we conclude with the idea

of integrating DA into HPC applications and evaluation of our

method against Spark.