MultiTune
Dynamic budget allocation for hyperparameter tuning
S. Dev (TU Delft - Electrical Engineering, Mathematics and Computer Science)
Lydia Y. Chen – Mentor (TU Delft - Data-Intensive Systems)
Neil Yorke-Smith – Graduation committee member (TU Delft - Algorithmics)
Jan S. Rellermeyer – Graduation committee member (TU Delft - Data-Intensive Systems)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
Hyperparameter optimization(HPO) forms a critical aspect for machine learning applications to attain superior performance. BOHB (Bayesian Optimization and HyperBand) is a state of the art HPO algorithm that approaches HPO in a multi-armed bandit strategy, augmented with Bayesian optimization to drive configuration sampling. However, BOHB requires predefined distribution of fidelities for each tuning task. The challenge in this is that it is impossible to define fidelities a priori, since each machine learning model is uniquely complex and requires different amount of compute resources for convergence. Furthermore, in our empirical analysis, we found that each HPO task rendered different performance trajectories on different fidelity (budget) types. Thus, the challenge of defining fidelities also extends to choosing an optimal budget type. To alleviate these challenges, we present MultiTune: a budget allocation scheme that builds on top of BOHB to dynamically define fidelities for optimization. MultiTune incorporates an algorithm to dynamically choose a preferred budget type for an HPO task, coupled with 2D gradient based budget constraint explorations to enable granular definition of fidelities. Through our empirical analysis, we show that MultiTune can consistently converge to a well performing configuration without significant computation overhead.