Online function minimization with convex random relu expansions

Conference paper (2017)

Authors

L. Bliek Team Raf Van de Plas - Mechanical, Maritime and Materials Engineering , Algorithmics - EEMCS
M.H.G. Verhaegen Team Michel Verhaegen - Mechanical, Maritime and Materials Engineering
S. Wahls Team Sander Wahls - Mechanical, Maritime and Materials Engineering
Research Group
Team Sander Wahls (Mechanical, Maritime and Materials Engineering) (TU Delft)
Deep learning Bayesian optimization Surrogate modeling Derivative-free optimization
More Info
expand_more

Published Date

2017

Language

English

Reuse Rights

Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.

Faculty

Mechanical, Maritime and Materials Engineering

Department

Delft Center for Systems and Control

Research Group

Team Sander Wahls

Abstract

We propose CDONE, a convex version of the DONE algorithm. DONE is a derivative-free online optimization algorithm that uses surrogate modeling with noisy measurements to find a minimum of objective functions that are expensive to evaluate. Inspired by their success in deep learning, CDONE makes use of rectified linear units, together with a nonnegativity constraint to enforce convexity of the surrogate model. This leads to a sparse and cheap to evaluate surrogate model of the unknown optimization objective that is still accurate and that can be minimized with convex optimization algorithms. The CDONE algorithm is demonstrated on a toy example and on the problem of hyper-parameter optimization for a deep learning example on handwritten digit classification.

Files

08168109.pdf
(.pdf | 0.266 Mb)