Metalearning HPO using Task Performance Representations

Abstract

Bayesian optimization for hyperparameter optimization (HPO) and combined algorithm selection and hyperparameter optimization (CASH) often relies on explicit hyperparameter encodings as surrogate inputs. In high-dimensional search spaces, however, such representations can make it difficult for standard surrogates to learn useful performance models. We instead study an alternative metalearning representation that describes each configuration directly as its historical performance across prior tasks and can be paired with standard surrogate models. We further apply this representation to maximal marginal relevance (MMR), a diversity-based re-ranking criterion from recommender systems, as a posterior-free acquisition rule. When selecting and optimising across multiple models on the TabArena benchmark, the performance-based representation paired with a linear Bayesian ridge regression model outperforms dedicated meta-learning baselines. On single-model HPO it remains a competitive second. Overall this method closely matches or improves on regret at a fraction of the complexity and cost, requiring no pre-training or tuning and scaling independently of search space dimensionality.