Solving ML with ML: Effectiveness of the Metropolis-Hastings algorithm for synthesizing Machine Learning Pipelines

Bachelor thesis (2023)

Authors

D.O. Sheremet Electrical Engineering, Mathematics and Computer Science

Contributors

S. Dumančić Algorithmics - EEMCS (supervisor 1)
T.R. Hinnerichs Algorithmics - EEMCS (supervisor 2)
D.M.J. Tax Pattern Recognition and Bioinformatics - EEMCS (supervisor 2)
Faculty
Electrical Engineering, Mathematics and Computer Science
Copyright: © 2023 Denys Sheremet
Machine Learning Program Synthesis AutoML Pipeline Synthesis Automated Machine Learning
More Info
expand_more

Published Date

25-06-2023

Language

English

Reuse Rights

Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.

Faculty

Electrical Engineering, Mathematics and Computer Science

Abstract

In AutoML, the search space of possible pipelines is often large and multidimensional. This makes it very important to use an efficient search algorithm. We measure the effectiveness of the Metropolis-Hastings algorithm (M-H) in a pipeline synthesis framework, when the search space is described by a context-free grammar. We also compare the performance of the M-H algorithm to other search algorithms. While AutoML frameworks use many different search algorithms, and comparisons between AutoML frameworks exist, this is the first paper that compares the performance of different search algorithms in the context of pipeline synthesis under equal conditions. We found that M-H is slightly outperformed by BFS2, the simplest possible search algorithm. We conclude that the datasets we use for evaluating the algorithms are too simple to meaningfully compare the performance of different search algorithms. We also conclude that for simple datasets, simple search algorithms work best.