Pipetune

Pipeline parallelism of hyper and system parameters tuning for deep learning clusters

Conference paper (2020)

Authors

Isabelly Rocha University of Neuchâtel
Nathaniel Morris Ohio State University
Lydia Y. Chen Data-Intensive Systems - EEMCS
Pascal Felber University of Neuchâtel
Robert Birke ABB Switzerland Ltd.
Valerio Schiavoni University of Neuchâtel
Research Group
Data-Intensive Systems (EEMCS) (TU Delft)
Parameter tuning Accuracy time trade-off Deep Neural Networks training
More Info
expand_more

Published Date

2020

Language

English

Reuse Rights

Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.

Faculty

Electrical Engineering, Mathematics and Computer Science

Department

Software Technology

Research Group

Data-Intensive Systems

Abstract

DNN learning jobs are common in today's clusters due to the advances in AI driven services such as machine translation and image recognition. The most critical phase of these jobs for model performance and learning cost is the tuning of hyperparameters. Existing approaches make use of techniques such as early stopping criteria to reduce the tuning impact on learning cost. However, these strategies do not consider the impact that certain hyperparameters and systems parameters have on training time. This paper presents PIPETUNE, a framework for DNN learning jobs that addresses the trade-offs between these two types of parameters. PIPETUNE takes advantage of the high parallelism and recurring characteristics of such jobs to minimize the learning cost via a pipelined simultaneous tuning of both hyper and system parameters. Our experimental evaluation using three different types of workloads indicates that PIPETUNE achieves up to 22.6% reduction and 1.7× speed up on tuning and training time, respectively. PipeTune not only improves performance but also lowers energy consumption up to 29%.