MultiTune

Abstract

Hyperparameter optimization(HPO) forms a critical aspect for machine learning applications to attain superior performance. BOHB (Bayesian Optimization and HyperBand) is a state of the art HPO algorithm that approaches HPO in a multi-armed bandit strategy, augmented with Bayesian optimization to drive configuration sampling. However, BOHB requires predefined distribution of fidelities for each tuning task. The challenge in this is that it is impossible to define fidelities a priori, since each machine learning model is uniquely complex and requires different amount of compute resources for convergence. Furthermore, in our empirical analysis, we found that each HPO task rendered different performance trajectories on different fidelity (budget) types. Thus, the challenge of defining fidelities also extends to choosing an optimal budget type. To alleviate these challenges, we present MultiTune: a budget allocation scheme that builds on top of BOHB to dynamically define fidelities for optimization. MultiTune incorporates an algorithm to dynamically choose a preferred budget type for an HPO task, coupled with 2D gradient based budget constraint explorations to enable granular definition of fidelities. Through our empirical analysis, we show that MultiTune can consistently converge to a well performing configuration without significant computation overhead.