10.1016/j.future.2011.05.026
https://zenodo.org/records/890130
oai:zenodo.org:890130
Taura, Kenjiro
Kenjiro
Taura
Matsuzaki, Takuya
Takuya
Matsuzaki
Miwa, Makoto
Makoto
Miwa
Kamoshida, Yoshikazu
Yoshikazu
Kamoshida
Yokoyama, Daisaku
Daisaku
Yokoyama
Dun, Nan
Nan
Dun
Shibata, Takeshi
Takeshi
Shibata
Jun, Choi Sung
Choi Sung
Jun
Tsujii, Jun'ichi
Jun'ichi
Tsujii
Design and implementation of GXP make — A workflow system based on make
Zenodo
2013
2013-02-01
Creative Commons Attribution Share Alike 4.0 International
This paper describes the rational behind designing
workflow systems based on the Unix make by showing a number
of idioms useful for workflows comprising many tasks. It also
demonstrates a specific design and implementation of such a
workflow system called GXP make. GXP make supports all
the features of GNU make and extends its platforms from
single node systems to clusters, clouds, supercomputers, and
distributed systems. Interestingly, it is achieved by a very small
code base that does not modify GNU make implementation at
all. While being not ideal for performance, it achieved a useful
performance and scalability of dispatching one million tasks in
approximately 16,000 seconds (60 tasks per second, including
dependence analysis) on an 8 core Intel Nehalem node. For
real applications, recognition and classification of protein-protein
interactions from biomedical texts on a supercomputer with more
than 8,000 cores are described.