Publication on the smart job scheduler implementation

Project: EuroScienceGateway was funded by the European Union programme Horizon Europe (HORIZON-INFRA-2021-EOSC-01-04) under grant agreement number 101057388 and by UK Research and Innovation (UKRI) under the UK government’s Horizon Europe funding guarantee grant number 10038963.

Document: D4.2 Publication on the smart job scheduler implementation

Work Package: Work Package 4. Building blocks for a sustainable operating model.

Task:

• Task 4.3 Implement a smart job-scheduling system across Europe

Lead Beneficiary: EGI

Contributing Beneficiary: ALU-FR, CESNET, EGI, UiO, and VIB

Executive Summary

Galaxy is currently using the Total Perspective Vortex (TPV) to schedule millions of jobs for hundred thousand users globally. While TPV has proven to be a robust meta-scheduling tool for Galaxy in the last years, there are areas of improvement that have been addressed in the EuroScienceGateway project:

1. Gathering live usage metrics from across the distributed computing endpoints connected to Galaxy in order to distribute the load across all sites.
2. Adding latitude and longitude attributes to data stores and computing endpoints to allocate jobs as close as possible to the location of the data.
3. Visualizing job distribution across sites with an intuitive dashboard.

As a result the EuroScienceGateway project has developed two new tools:

1. TPV Broker for the efficient meta-scheduling of jobs taking into account real-time usage metrics and data-locality information
2. Galaxy Job Radar: a web dashboard to easily visualize the allocation of jobs across all sites

The EuroScienceGateway project has significantly improved the meta-scheduling of jobs for Galaxy, resulting in less waiting times for users to see their job completed and improving resource utilization across all sites.