Conference paper Open Access

Learning Workflow Scheduling on Multi-Resource Clusters

Hu, Yang; de Laat, Cees; Zhao, Zhiming


MARC21 XML Export

<?xml version='1.0' encoding='UTF-8'?>
<record xmlns="http://www.loc.gov/MARC21/slim">
  <leader>00000nam##2200000uu#4500</leader>
  <datafield tag="653" ind1=" " ind2=" ">
    <subfield code="a">Workflow Scheduling</subfield>
  </datafield>
  <datafield tag="653" ind1=" " ind2=" ">
    <subfield code="a">Multi-resource Clusters</subfield>
  </datafield>
  <datafield tag="653" ind1=" " ind2=" ">
    <subfield code="a">Directed-Acyclic Graph (DAG)</subfield>
  </datafield>
  <datafield tag="653" ind1=" " ind2=" ">
    <subfield code="a">Deep Reinforcement Learning</subfield>
  </datafield>
  <controlfield tag="005">20200120172844.0</controlfield>
  <controlfield tag="001">3466676</controlfield>
  <datafield tag="711" ind1=" " ind2=" ">
    <subfield code="d">15-17 August 2019</subfield>
    <subfield code="g">NAS</subfield>
    <subfield code="a">The 14th International Conference on Networking, Architecture, and Storage (NAS 2019)</subfield>
    <subfield code="c">Enshi, China</subfield>
  </datafield>
  <datafield tag="700" ind1=" " ind2=" ">
    <subfield code="u">University of Amsterdam</subfield>
    <subfield code="a">de Laat, Cees</subfield>
  </datafield>
  <datafield tag="700" ind1=" " ind2=" ">
    <subfield code="u">University of Amsterdam</subfield>
    <subfield code="0">(orcid)0000-0002-6717-9418</subfield>
    <subfield code="a">Zhao, Zhiming</subfield>
  </datafield>
  <datafield tag="856" ind1="4" ind2=" ">
    <subfield code="s">390857</subfield>
    <subfield code="z">md5:b9cc45d788adf288491267ffaf6bded9</subfield>
    <subfield code="u">https://zenodo.org/record/3466676/files/2019.8.conference.nas.camera.pdf</subfield>
  </datafield>
  <datafield tag="542" ind1=" " ind2=" ">
    <subfield code="l">open</subfield>
  </datafield>
  <datafield tag="856" ind1="4" ind2=" ">
    <subfield code="y">Conference website</subfield>
    <subfield code="u">http://www.nas-conference.org/NAS-2019/</subfield>
  </datafield>
  <datafield tag="260" ind1=" " ind2=" ">
    <subfield code="c">2019-08-16</subfield>
  </datafield>
  <datafield tag="909" ind1="C" ind2="O">
    <subfield code="p">openaire</subfield>
    <subfield code="o">oai:zenodo.org:3466676</subfield>
  </datafield>
  <datafield tag="100" ind1=" " ind2=" ">
    <subfield code="u">University of Amsterdam</subfield>
    <subfield code="a">Hu, Yang</subfield>
  </datafield>
  <datafield tag="245" ind1=" " ind2=" ">
    <subfield code="a">Learning Workflow Scheduling on Multi-Resource Clusters</subfield>
  </datafield>
  <datafield tag="536" ind1=" " ind2=" ">
    <subfield code="c">676247</subfield>
    <subfield code="a">A Europe-wide Interoperable Virtual Research Environment to Empower Multidisciplinary Research Communities and Accelerate Innovation and Collaboration</subfield>
  </datafield>
  <datafield tag="536" ind1=" " ind2=" ">
    <subfield code="c">654182</subfield>
    <subfield code="a">Environmental Research Infrastructures Providing Shared Solutions for Science and Society</subfield>
  </datafield>
  <datafield tag="536" ind1=" " ind2=" ">
    <subfield code="c">643963</subfield>
    <subfield code="a">Software Workbench for Interactive, Time Critical and Highly self-adaptive cloud applications</subfield>
  </datafield>
  <datafield tag="536" ind1=" " ind2=" ">
    <subfield code="c">825134</subfield>
    <subfield code="a">smART socIal media eCOsytstem in a blockchaiN Federated environment</subfield>
  </datafield>
  <datafield tag="540" ind1=" " ind2=" ">
    <subfield code="u">https://creativecommons.org/licenses/by/4.0/legalcode</subfield>
    <subfield code="a">Creative Commons Attribution 4.0 International</subfield>
  </datafield>
  <datafield tag="650" ind1="1" ind2="7">
    <subfield code="a">cc-by</subfield>
    <subfield code="2">opendefinition.org</subfield>
  </datafield>
  <datafield tag="520" ind1=" " ind2=" ">
    <subfield code="a">&lt;p&gt;Workflow scheduling is one of the key issues in&lt;/p&gt;

&lt;p&gt;the management of workflow execution. Typically, a workflow&lt;/p&gt;

&lt;p&gt;application can be modeled as a Directed-Acyclic Graph (DAG).&lt;/p&gt;

&lt;p&gt;In this paper, we present GoDAG, an approach that can learn&lt;/p&gt;

&lt;p&gt;to well schedule workflows on multi-resource clusters. GoDAG&lt;/p&gt;

&lt;p&gt;directly learns the scheduling policy from experience through&lt;/p&gt;

&lt;p&gt;deep reinforcement learning. In order to adapt deep reinforcement&lt;/p&gt;

&lt;p&gt;learning methods, we propose a novel state representation,&lt;/p&gt;

&lt;p&gt;a practical action space and a corresponding reward definition&lt;/p&gt;

&lt;p&gt;for workflow scheduling problem. We implement a GoDAG&lt;/p&gt;

&lt;p&gt;prototype and a simulator to simulate task running on multiresource&lt;/p&gt;

&lt;p&gt;clusters. In the evaluation, we compare the GoDAG with&lt;/p&gt;

&lt;p&gt;three state-of-the-art heuristics. The results show that GoDAG&lt;/p&gt;

&lt;p&gt;outperforms the baseline heuristics, leading to less average&lt;/p&gt;

&lt;p&gt;makespan to different workflow structures.&lt;/p&gt;</subfield>
  </datafield>
  <datafield tag="024" ind1=" " ind2=" ">
    <subfield code="a">10.1109/NAS.2019.8834720</subfield>
    <subfield code="2">doi</subfield>
  </datafield>
  <datafield tag="980" ind1=" " ind2=" ">
    <subfield code="a">publication</subfield>
    <subfield code="b">conferencepaper</subfield>
  </datafield>
</record>
46
43
views
downloads
Views 46
Downloads 43
Data volume 16.8 MB
Unique views 43
Unique downloads 40

Share

Cite as