Conference paper Open Access

Learning Workflow Scheduling on Multi-Resource Clusters

Hu, Yang; de Laat, Cees; Zhao, Zhiming

Workflow scheduling is one of the key issues in

the management of workflow execution. Typically, a workflow

application can be modeled as a Directed-Acyclic Graph (DAG).

In this paper, we present GoDAG, an approach that can learn

to well schedule workflows on multi-resource clusters. GoDAG

directly learns the scheduling policy from experience through

deep reinforcement learning. In order to adapt deep reinforcement

learning methods, we propose a novel state representation,

a practical action space and a corresponding reward definition

for workflow scheduling problem. We implement a GoDAG

prototype and a simulator to simulate task running on multiresource

clusters. In the evaluation, we compare the GoDAG with

three state-of-the-art heuristics. The results show that GoDAG

outperforms the baseline heuristics, leading to less average

makespan to different workflow structures.

Files (390.9 kB)
Name Size
2019.8.conference.nas.camera.pdf
md5:b9cc45d788adf288491267ffaf6bded9
390.9 kB Download
5
5
views
downloads
Views 5
Downloads 5
Data volume 2.0 MB
Unique views 3
Unique downloads 3

Share

Cite as