Published February 1, 2013
| Version v1
Journal article
Open
Design and implementation of GXP make — A workflow system based on make
Description
This paper describes the rational behind designing
workflow systems based on the Unix make by showing a number
of idioms useful for workflows comprising many tasks. It also
demonstrates a specific design and implementation of such a
workflow system called GXP make. GXP make supports all
the features of GNU make and extends its platforms from
single node systems to clusters, clouds, supercomputers, and
distributed systems. Interestingly, it is achieved by a very small
code base that does not modify GNU make implementation at
all. While being not ideal for performance, it achieved a useful
performance and scalability of dispatching one million tasks in
approximately 16,000 seconds (60 tasks per second, including
dependence analysis) on an 8 core Intel Nehalem node. For
real applications, recognition and classification of protein-protein
interactions from biomedical texts on a supercomputer with more
than 8,000 cores are described.
Files
article.pdf
Files
(843.6 kB)
Name | Size | Download all |
---|---|---|
md5:3259da71febed6c900500db90a5a59a7
|
843.6 kB | Preview Download |