Published June 4, 2014 | Version v1
Working paper Open

Scaling of Biological Data Work ows to Large HPC Systems - A Case Study in Marine Genomics -

  • 1. Department for Research Computing, University Center for Information Technology (USIT), University of Oslo, P.O. Box 1059, Blindern, 0316 Oslo, Norway
  • 1. Department for Research Computing, University Center for Information Technology (USIT), University of Oslo, P.O. Box 1059, Blindern, 0316 Oslo, Norway
  • 2. Center for Ecological and Evolutionary Synthesis, Department of Biosciences (CEES), University of Oslo, P.O. Box 1066, Blindern, 0316 Oslo, Norway

Description

Sequencing projects, like the Aqua Genome project, generate vast amounts of data which is processed through dif-
ferent work ows composed of several steps linked together. Currently, such workflows are often run manually on
large servers. With the increasing amount of raw data that approach is no longer feasible. The successful imple-
mentation of the project's goals requires 2-3 orders of magnitude scaling of computing, while achieving high reli-
ability on and supporting ease-of-use of super computing resources at the same time. We describe two example
use cases, the implementation challenges and constraints, the actual application enabling and report our ndings.

Files

WP171.pdf

Files (504.4 kB)

Name Size Download all
md5:50508e75466ab6fbd53d3204d172b794
504.4 kB Preview Download

Additional details

Funding

PRACE-3IP – PRACE - Third Implementation Phase Project 312763
European Commission