Enabling workflow aware scheduling on HPC systems
(English)Manuscript (preprint) (Other academic)
Workows from diverse scientic domains are increasingly present in the workloads of current HPC systems. However, HPC scheduling systems do not incorporate workow specic mechanisms beyond the capacity to declare dependencies between jobs. us, when users run workows as sets of batch jobs with completion dependencies, the workows experience long turn around times. Alternatively, when they are submied as single jobs, allocating the maximum requirementof resources for the whole runtime, they resources, reducing the HPC system utilization. In this paper, we present a workow aware scheduling (WoAS) system that enables pre-existing scheduling algorithms to take advantage of the ne grained workow resource requirements and structure, without any modication to the original algorithms. e current implementation of WoAS is integrated in Slurm, a widely used HPC batch scheduler. We evaluate the system in simulation using real and synthetic workows and a synthetic baseline workload that captures the job paerns observed over three years of the real workload data of Edison, a large supercomputer hosted at the National Energy Research Scientic Computing Center. Finally, our results show that WoAS eectively reduces workow turnaround time and improves system utilization without a signicant impact on the slowdown of traditional jobs.
scheduling, workflows, HPC, supercomputing, High Performance Computing
Research subject Computing Science
IdentifiersURN: urn:nbn:se:umu:diva-132982OAI: oai:DiVA.org:umu-132982DiVA: diva2:1084852
FundereSSENCE - An eScience CollaborationSwedish Research Council, C0590801