Conference paper Open Access

Learning Workflow Scheduling on Multi-Resource Clusters

Hu, Yang; de Laat, Cees; Zhao, Zhiming

Workflow scheduling is one of the key issues in

the management of workflow execution. Typically, a workflow

application can be modeled as a Directed-Acyclic Graph (DAG).

In this paper, we present GoDAG, an approach that can learn

to well schedule workflows on multi-resource clusters. GoDAG

directly learns the scheduling policy from experience through

deep reinforcement learning. In order to adapt deep reinforcement

learning methods, we propose a novel state representation,

a practical action space and a corresponding reward definition

for workflow scheduling problem. We implement a GoDAG

prototype and a simulator to simulate task running on multiresource

clusters. In the evaluation, we compare the GoDAG with

three state-of-the-art heuristics. The results show that GoDAG

outperforms the baseline heuristics, leading to less average

makespan to different workflow structures.

Files (390.9 kB)
Name Size
2019.8.conference.nas.camera.pdf
md5:b9cc45d788adf288491267ffaf6bded9
390.9 kB Download
32
40
views
downloads
Views 32
Downloads 40
Data volume 15.6 MB
Unique views 29
Unique downloads 37

Share

Cite as