umu.sePublications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
HPC scheduling in a brave new world
Umeå University, Faculty of Science and Technology, Department of Computing Science. (Distributed Systems)ORCID iD: 0000-0003-3315-8253
2017 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

Many breakthroughs in scientific and industrial research are supported by simulations and calculations performed on high performance computing (HPC) systems. These systems typically consist of uniform, largely parallel compute resources and high bandwidth concurrent file systems interconnected by low latency synchronous networks. HPC systems are managed by batch schedulers that order the execution of application jobs to maximize utilization while steering turnaround time. In the past, demands for greater capacity were met by building more powerful systems with more compute nodes, greater transistor densities, and higher processor operating frequencies. Unfortunately, the scope for further increases in processor frequency is restricted by the limitations of semiconductor technology. Instead, parallelism within processors and in numbers of compute nodes is increasing, while the capacity of single processing units remains unchanged. In addition, HPC systems’ memory and I/O hierarchies are becoming deeper and more complex to keep up with the systems’ processing power. HPC applications are also changing: the need to analyze large data sets and simulation results is increasing the importance of data processing and data-intensive applications. Moreover, composition of applications through workflows within HPC centers is becoming increasingly important. This thesis addresses the HPC scheduling challenges created by such new systems and applications. It begins with a detailed analysis of the evolution of the workloads of three reference HPC systems at the National Energy Research Supercomputing Center (NERSC), with a focus on job heterogeneity and scheduler performance. This is followed by an analysis and improvement of a fairshare prioritization mechanism for HPC schedulers. The thesis then surveys the current state of the art and expected near-future developments in HPC hardware and applications, and identifies unaddressed scheduling challenges that they will introduce. These challenges include application diversity and issues with workflow scheduling or the scheduling of I/O resources to support applications. Next, a cloud-inspired HPC scheduling model is presented that can accommodate application diversity, takes advantage of malleable applications, and enables short wait times for applications. Finally, to support ongoing scheduling research, an open source scheduling simulation framework is proposed that allows new scheduling algorithms to be implemented and evaluated in a production scheduler using workloads modeled on those of a real system. The thesis concludes with the presentation of a workflow scheduling algorithm to minimize workflows’ turnaround time without over-allocating resources.

Place, publisher, year, edition, pages
Umeå: Umeå universitet , 2017. , 122 p.
Series
Report / UMINF, ISSN 0348-0542 ; 17.05
Keyword [en]
High Performance Computing, HPC, supercomputing, scheduling, workflows, workloads, exascale
National Category
Computer Science
Research subject
Computing Science
Identifiers
URN: urn:nbn:se:umu:diva-132983ISBN: 978-91-7601-693-0 (print)OAI: oai:DiVA.org:umu-132983DiVA: diva2:1084867
Public defence
2017-04-21, MA121, MIT-Huset, Umeå Universitet, Umeå, 10:15 (English)
Opponent
Supervisors
Funder
eSSENCE - An eScience CollaborationSwedish Research Council, C0590801EU, Horizon 2020, 610711EU, FP7, Seventh Framework Programme, 732667
Note

Work also supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research (ASCR) and we used resources at the National Energy Research Scientific Computing Center, a DOE Office of Science User Facility, supported by the Officece of Science of the U.S. Department of Energy, both under Contract No. DE-AC02-05CH11231.

Available from: 2017-03-29 Created: 2017-03-27 Last updated: 2017-05-29Bibliographically approved
List of papers
1. Towards understanding HPC users and systems: a NERSC case study
Open this publication in new window or tab >>Towards understanding HPC users and systems: a NERSC case study
Show others...
(English)Manuscript (preprint) (Other academic)
Abstract [en]

The high performance computing (HPC) scheduling landscape is changing. Previously dominated by tightly coupled MPI jobs, HPC workloads are increasingly including high-throughput, data-intensive, and stream-processing applications. As a consequence, workloads are becoming more diverse at both application and job level, posing new challenges to classical HPC schedulers. There is a need to understand the current HPC workloads and their evolution towards the future in order to perform informed scheduling research and enable efficient scheduling in future HPC systems. In this paper, we present a methodology to characterize workloads and asses their heterogeneity, both for a particular time period and as they evolve over time. We apply this methodology to the workloads of three systems (Hopper, Edison, and Carver) at the National Energy Research Scientific Computing Center (NERSC). We present the resulting characterization of jobs, queues, heterogeneity, and performance that includes detailed information of a year of workload (2014) and evolution through the systems’ lifetime. Among the results, we highlight the observation of discontinuities in the jobs’ wait time for priority groups with high job diversity. Finally, we conclude by summarizing our analysis to establish a reference and inform future scheduling research.

Keyword
workload analysis, supercomputer, HPC, scheduling, NERSC, heterogeneity, k-means
National Category
Computer Science
Research subject
Computing Science
Identifiers
urn:nbn:se:umu:diva-132980 (URN)
Funder
eSSENCE - An eScience CollaborationEU, Horizon 2020, 610711EU, Horizon 2020, 732667Swedish Research Council, C0590801
Note

Work also supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research (ASCR) and we used resources at the National Energy Research Scientific Computing Center, a DOE Office of Science User Facility, supported by the Officece of Science of the U.S. Department of Energy, both under Contract No. DE-AC02-05CH11231.

Available from: 2017-03-27 Created: 2017-03-27 Last updated: 2017-05-29
2. Priority operators for fairshare scheduling
Open this publication in new window or tab >>Priority operators for fairshare scheduling
2015 (English)In: Job scheduling strategies for parallel processing (JSSPP 2014), 2015, 70-89 p.Conference paper, (Refereed)
Abstract [en]

Collaborative resource sharing in distributed computing requires scalable mechanisms for allocation and control of user quotas. Decentralized fairshare prioritization is a technique for enforcement of user quotas that can be realized without centralized control. The technique is based on influencing the job scheduling order of local resource management systems using an algorithm that establishes a semantic for prioritization of jobs based on the individual distances between user's quota allocations and user's historical resource usage (i.e. intended and current system state). This work addresses the design and evaluation of priority operators, mathematical functions to quantify fairshare distances, and identify a set of desirable characteristics for fairshare priority operators. In addition, this work also proposes a set of operators for fairshare prioritization, establishes a methodology for verification and evaluation of operator characteristics, and evaluates the proposed operator set based on this mathematical framework. Limitations in the numerical representation of scheduling factor values are identified as a key challenge in priority operator formulation, and it is demonstrated that the contributed priority operators (the Sigmoid operator family) behave robustly even in the presence of severe resolution limitations.

Series
Lecture Notes in Computer Science, ISSN 0302-9743 ; 8828
National Category
Computer Science
Identifiers
urn:nbn:se:umu:diva-106517 (URN)10.1007/978-3-319-15789-4_5 (DOI)000355729800005 ()978-3-319-15788-7 (ISBN)978-3-319-15789-4 (ISBN)
Conference
18th International Workshop on Job Scheduling Strategies for Parallel Processing (JSSPP), MAY 23, 2014, Phoenix, AZ
Available from: 2015-07-15 Created: 2015-07-14 Last updated: 2017-03-28Bibliographically approved
3. A2L2: an application aware flexible HPC scheduling model for low-latency allocation
Open this publication in new window or tab >>A2L2: an application aware flexible HPC scheduling model for low-latency allocation
2015 (English)In: VTDC '15: proceedings of the 8th International workshop on virtualization technologies in distributed computing, ACM Digital Library, 2015, , 11-19 p.11-19 p.Conference paper, (Refereed)
Abstract [en]

High-performance computing (HPC) is focused on providing large-scale compute capacity to scientific applications. HPC schedulers tend to be optimized for large parallel batch jobs and, as such, often overlook the requirements of other scientific applications. In this work, we propose a cloud-inspired HPC scheduling model that aims to capture application performance and requirement models (Application Aware - A2) and dynamically resize malleable application resource allocations to be able to support applications with critical performance or deadline requirements. (Low Latency allocation - L2). The proposed model incorporates measures to improve data-intensive applications performance on HPC systems and is derived from a set of cloud scheduling techniques that are identified as applicable in HPC environments. The model places special focus on dynamically malleable applications; data-intensive applications that support dynamic resource allocation without incurring severe performance penalties; which are proposed for fine-grained back-filling and dynamic resource allocation control without job preemption.

Place, publisher, year, edition, pages
ACM Digital Library, 2015. 11-19 p.
Keyword
Scheduling, job, HPC, malleable, applications, low-latency
National Category
Computer Systems
Research subject
Computing Science
Identifiers
urn:nbn:se:umu:diva-110526 (URN)10.1145/2755979.2755983 (DOI)978-1-4503-3573-7 (ISBN)
Conference
8th International Workshop on Virtualization Technologies in Distributed Computing (VTDC), Portland, Oregon, June 15-16, 2015.
Funder
eSSENCE - An eScience CollaborationEU, FP7, Seventh Framework Programme, 610711Swedish Research Council, C0590801
Note

Work also supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research (ASCR) and we used resources at the National Energy Research Scientific Computing Center, a DOE Office of Science User Facility, supported by the Officece of Science of the U.S. Department of Energy, both under Contract No. DE-AC02-05CH11231.

Available from: 2015-10-22 Created: 2015-10-22 Last updated: 2017-05-29Bibliographically approved
4. ScSF: a scheduling simulation framework
Open this publication in new window or tab >>ScSF: a scheduling simulation framework
2017 (English)In: Proceedings of the 21th Workshop on Job Scheduling Strategies for Parallel Processing, 2017Conference paper, (Refereed)
Abstract [en]

High-throughput and data-intensive applications are increasingly present, often composed as workflows, in the workloads of current HPC systems. At the same time, trends for future HPC systems point towards more heterogeneous systems with deeper I/O and memory hierarchies. However, current HPC schedulers are designed to support classical large tightly coupled parallel jobs over homogeneous systems. Therefore, There is an urgent need to investigate new scheduling algorithms that can manage the future workloads on HPC systems. However, there is a lack of appropriate models and frameworks to enable development, testing, and validation of new scheduling ideas. In this paper, we present an open-source scheduler simulation framework (ScSF) that covers all the steps of scheduling research through simulation. ScSF provides capabilities for workload modeling, workload generation, system simulation, comparative workload analysis, and experiment orchestration. The simulator is designed to be run over a distributed computing infrastructure enabling to test at scale. We describe in detail a use case of ScSF to develop new techniques to manage scientific workflows in a batch scheduler. In the use case, such technique was implemented in the framework scheduler. For evaluation purposes, 1728 experiments, equivalent to 33 years of simulated time, were run in a deployment of ScSF over a distributed infrastructure of 17 compute nodes during two months. Finally, the experimental results were analyzed in the framework to judge that the technique minimizes workflows’ turnaround time without over-allocating resources. Finally, we discuss lessons learned from our experiences that will help future researchers.

Series
Lecture Notes in Computer Science, ISSN 0302-9743
Keyword
slurm, simulation, scheduling, HPC, High Performance Computing, workload, generation, analysis
National Category
Computer Science
Research subject
Computer Science
Identifiers
urn:nbn:se:umu:diva-132981 (URN)
Conference
21th Workshop on Job Scheduling Strategies for Parallel Processing (JSSP 2017), Orlando FL, USA, June 2nd, 2017.
Funder
eSSENCE - An eScience CollaborationSwedish Research Council, C0590801
Note

Work also supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research (ASCR) and we used resources at the National Energy Research Scientific Computing Center, a DOE Office of Science User Facility, supported by the Officece of Science of the U.S. Department of Energy, both under Contract No. DE-AC02-05CH11231.

Available from: 2017-03-27 Created: 2017-03-27 Last updated: 2017-05-29
5. Enabling workflow aware scheduling on HPC systems
Open this publication in new window or tab >>Enabling workflow aware scheduling on HPC systems
(English)Manuscript (preprint) (Other academic)
Abstract [en]

Workƒows from diverse scienti€c domains are increasingly present in the workloads of current HPC systems. However, HPC scheduling systems do not incorporate workƒow speci€c mechanisms beyond the capacity to declare dependencies between jobs. Œus, when users run workƒows as sets of batch jobs with completion dependencies, the workƒows experience long turn around times. Alternatively, when they are submiŠed as single jobs, allocating the maximum requirementof resources for the whole runtime, they resources, reducing the HPC system utilization. In this paper, we present a workƒow aware scheduling (WoAS) system that enables pre-existing scheduling algorithms to take advantage of the €ne grained workƒow resource requirements and structure, without any modi€cation to the original algorithms. Œe current implementation of WoAS is integrated in Slurm, a widely used HPC batch scheduler. We evaluate the system in simulation using real and synthetic workƒows and a synthetic baseline workload that captures the job paŠerns observed over three years of the real workload data of Edison, a large supercomputer hosted at the National Energy Research Scienti€c Computing Center. Finally, our results show that WoAS e‚ectively reduces workƒow turnaround time and improves system utilization without a signi€cant impact on the slowdown of traditional jobs.

Keyword
scheduling, workflows, HPC, supercomputing, High Performance Computing
National Category
Computer Science
Research subject
Computing Science
Identifiers
urn:nbn:se:umu:diva-132982 (URN)
Funder
eSSENCE - An eScience CollaborationSwedish Research Council, C0590801
Note

Work also supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research (ASCR) and we used resources at the National Energy Research Scientific Computing Center, a DOE Office of Science User Facility, supported by the Officece of Science of the U.S. Department of Energy, both under Contract No. DE-AC02-05CH11231.

Available from: 2017-03-27 Created: 2017-03-27 Last updated: 2017-05-29

Open Access in DiVA

fulltext(7773 kB)132 downloads
File information
File name FULLTEXT01.pdfFile size 7773 kBChecksum SHA-512
3a04f23d63b5492879a0fc4e236cdb5ef880562e83860928f0ba585947f277a6fa2ddcf2fc5295e5db2a97b4775094474925bebb6eb1675c3fea2337537961b1
Type fulltextMimetype application/pdf
spikblad(74 kB)20 downloads
File information
File name SPIKBLAD01.pdfFile size 74 kBChecksum SHA-512
3192508ce7f3641676312558050a47560364cf7821993093f6b66f12d42fcfa8ce14d501e3de62a47f465f821dc5d164c5df9fbb4f8295763ac8c71daccbbc9c
Type spikbladMimetype application/pdf
omslag(1605 kB)6 downloads
File information
File name COVER01.pdfFile size 1605 kBChecksum SHA-512
992785db7b1efafa5073fff2c054b6b6eb1c1d684d10adb22d5311fb3647a7421a7f33d55dab44a7ea268679c6eb3f2b02c438195bbbba5012fbdf6b6cf45d48
Type coverMimetype application/pdf

Search in DiVA

By author/editor
Gonzalo P., Rodrigo
By organisation
Department of Computing Science
Computer Science

Search outside of DiVA

GoogleGoogle Scholar
Total: 132 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

Total: 3873 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf