Modeling Inference Time of Deep Neural Networks on Memory-constrained Systems

Modeling Inference Time of Deep Neural Networks on Memory-constrained Systems

Brouwer, Hans (TU Delft Electrical Engineering, Mathematics and Computer Science)

Chen, Lydia Y. (mentor)
Ghiassi, S. (graduation committee)
Cox, B.A. (graduation committee)
Zuniga, Marco (graduation committee)

Delft University of Technology

Computer Science and Engineering

CSE3000 Research Project

2020-06-22

Deep neural networks have revolutionized multiple fields within computer science. It is important to have a comprehensive understanding of the memory requirements and performance of deep networks on low-resource systems. While there have been efforts to this end, the effects of severe memory limits and heavy swapping are understudied. We have profiled multiple deep networks under varying memory restrictions and on different hardware. Using this data, we develop two modeling approaches to predict the execution time of a network based on a description of its layers and the available memory. The first modeling approach is based on engineering predictive features through a theoretical analysis of the computations required to execute a layer. The second approach uses a LASSO regression to select predictive features from an expanded set of predictors. Both approaches achieve a mean absolute percentage error of 5% on log-transformed data, but suffer degraded performance on transformation of predictions back to regular space.

Performance Analysis
Deep Neural Networks
Inference
Memory-constrained
Machine Learning
Prediction
Modeling

http://resolver.tudelft.nl/uuid:bcfb9bb6-a21e-4f0b-b98d-5410e399ff34

Student theses

bachelor thesis

© 2020 Hans Brouwer