Black Magic in Deep Learning

Abstract

Deep learning is proving to be a useful tool in solving problems from various domains. Despite a rich research activity leading to numerous interesting deep learning models, recent large scale studies have shown that with hyperparameter optimization it is hard to distinguish these models based on their final performance. Hyperparameter optimization has shown to improve the state of the art results on several occasions. These results cast the doubts over the performance of these improved deep learning models and lead to the question whether the final performance of a deep learning model is dependent on the person performing the hyperparameter optimization task. A user study was conducted to evaluate the impact of human's prior experience in deep learning on the final performance of a deep learning model. 31 people with different levels of experience in deep learning were invited to perform a hyperparameter optimization task. The collected data was analyzed to find the relationship between human and the final performance of the deep learning model used for the user study. From the results, we observed that the final performance of the model vary with every participant, and a strong correlation between the participant's experience and the final performance achieved. Our data suggest that an experienced participant finds better results using fewer resources.