A Framework for Developing Next-Generation HPC Schedulers

Description

Abstract (English)

Introduction: The increasing diversity and scale of high-performance computing systems and workloads, ranging from traditional scientific simulations to emerging AI/ML applications, place growing demands on resource management. As systems continue to scale in both size and complexity, scheduling strategies must evolve to balance multiple, often competing objectives such as throughput, energy efficiency, and user satisfaction. Techniques such as co-scheduling and moldability offer immense potential gains. Validating new scheduling algorithms is challenging, as implementing them in existing simulators is often cumbersome and time-consuming for researchers. Real-world testing is essential but rarely feasible, as researchers lack dedicated clusters under administrative privileges. Moreover, large-scale experiments are costly in time, computation, and energy, making selective experimentation necessary.

We propose a unified evaluation framework that eliminates these barriers. By combining ELiSE (Efficient Lightweight Scheduling Estimator) and Flurm (Flux + Slurm), we provide a closed-loop environment where researchers can Design, Simulate, Test, and Deploy custom scheduling policies, so as to comprehensively and realistically examine, compare, and evaluate their performance on various key metrics.

The Framework Architecture The framework follows a four-stage cycle, beginning with algorithm conception and concluding with its evaluation in a realistic HPC environment.

Design: The cycle begins with the conceptualization of advanced scheduling strategies to address modern HPC challenges. The framework is specifically designed to support complex paradigms like Co-Scheduling (running jobs on shared resources to boost throughput) and Job Moldability (determining job process counts at schedule time rather than submission time), which are traditionally difficult to validate. Beyond these, the framework is extensible, allowing the introduction and evaluation of other advanced scheduling algorithms, including user-defined and intelligent strategies.

Simulate (ELiSE): Before real-world testing, researchers use ELiSE (Efficient Lightweight Scheduling Estimator), a Python-based prototyping environment, to simulate conceived algorithms. Unlike rigid C-based schedulers, ELiSE allows complex algorithms to be built by extending simple scheduler classes with high-level Python abstractions. It supports GUI and CLI interfaces, integrates easily with notebooks for custom analysis, and enables batch experiments. ELiSE can replay various workload logs (e.g. Standard Workload Format (SWF)), or generate synthetic traffic to evaluate algorithms on metrics such as Makespan, Turnaround Time, System Utilization, and Fairness.

Integrate: Upon achieving satisfactory performance metrics in the simulation phase, the researcher transitions to the implementation stage by porting the scheduling logic into the Flux next-generation resource manager, developed by LLNL and used by the number-one-ranked TOP500 supercomputer El Capitan. Validity testing can be performed within a controlled Docker environment.

Deploy: The key innovation of our framework is Flurm, which requests a standard batch multi-node allocation from the host Slurm system as a typical job and bootstraps a user-space Flux instance inside it. This enables schedulers validated in simulation to manage real MPI jobs in isolation, experiencing real network and memory contention without administrative intervention. Experiments show that Flurm incurs minimal overhead, making it a reliable indicator of real-world scheduler behavior.

Key Use Cases Our poster highlights two complex scenarios where this framework proves essential, aside from traditional HPC scheduling:

Co-Scheduling: In traditional HPC systems, each application is executed isolated from other applications (compact execution), and utilizes all cores on each allocated node. In contrast, co-scheduling environments necessitate running applications in spread mode, wherein each application is allocated only half the cores per node and distributed across twice the number of nodes compared to compact execution. The remaining cores in each node are then occupied by other jobs that are also running in spread mode. Co-scheduling improves HPC performance by boosting memory-bound applications through effective job co-location, without harming compute- or communication-bound workloads, resulting in higher application speedups and overall system throughput.

Job Moldability: Submitting a job with a large process count can reduce its execution time if the job scales well. However, it may also increase its wait time if sufficient resources are not available. Choosing the optimal process count can be difficult for users who are unaware of current system usage or the scalability of their job. Moldable jobs address this by allowing users to submit a job with multiple possible process counts, leaving it to the scheduler to select the most appropriate one at scheduling time. Our framework enables the development of schedulers that dynamically select the optimal job size for each queued job, using heuristics and considering a variety of factors such as job scalability and system and/or queue load.

ELiSE includes several co-scheduling and moldable job algorithms, with hooks to develop more. Similarly, Flux has been extended to support these features, with multiple algorithms already implemented.

Conclusion: This framework significantly advances HPC scheduling research by bridging simulation and real-system validation through a unified pipeline. By combining ELiSE and Flurm, researchers can prototype, test, and deploy scheduling algorithms without administrative privileges or dedicated clusters. It also enables practical experimentation with advanced techniques such as co-scheduling and moldable jobs, which are otherwise difficult to evaluate. Overall, the framework lowers the barrier to entry, accelerates development, and provides an accurate, efficient environment for assessing HPC scheduling policies.

Acknowledgment: This work has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 101177590 (Project SEANERGYS - HORIZON-EUROHPC-JU-2023-ENERGY-04) and by the Greek General Secretariat of Research and Innovation (GSRI).

References Karapanagiotis, E., Triantafyllis, N., Tsoukleidis-Karydakis, A., Goumas, G., Koziris, N. (2026). ELiSE: A Tool to Support Algorithmic Design for HPC Co-scheduling. In: Klusáček, D., Corbalán, J., Rodrigo, G.P. (eds) Job Scheduling Strategies for Parallel Processing. JSSPP 2025. Lecture Notes in Computer Science, vol 16210. Springer, Cham. https://doi.org/10.1007/978-3-032-10507-3_16

Tsoukleidis-Karydakis, A., Karapanagiotis, E., Triantafyllis, N., Koziris, N., Goumas, G. (2026). Performance Models to Support HPC Co-scheduling. In: Klusáček, D., Corbalán, J., Rodrigo, G.P. (eds) Job Scheduling Strategies for Parallel Processing. JSSPP 2025. Lecture Notes in Computer Science, vol 16210. Springer, Cham. https://doi.org/10.1007/978-3-032-10507-3_15

Triantafyllis, N., Tsoukleidis-Karydakis, A., Karapanagiotis, E., Kellari, M., Goumas, G., & Koziris, N. (2025). Outlining an HPC Co-Scheduler. IEEE International Parallel & Distributed Processing Symposium, PhD Forum - Poster.

Triantafyllis, N., Karapanagiotis, E., Papavasileiou, A., Goumas, G., & Koziris, N. (2024). Practicing the benefits of co-execution in HPC. EuroSys - Poster.