Title
Uncovering Energy-Efficient Practices in Deep Learning Training: Preliminary Steps Towards Green AI
Author
Yarally, Tim (Student TU Delft)
Cruz, Luis (TU Delft Software Engineering) 
Feitosa, Daniel (Rijksuniversiteit Groningen)
Sallou, J. (TU Delft Software Engineering)
van Deursen, A. (TU Delft Software Technology)
Department
Software Technology
Date
2023
Abstract
Modern AI practices all strive towards the same goal: better results. In the context of deep learning, the term "results"often refers to the achieved accuracy on a competitive problem set. In this paper, we adopt an idea from the emerging field of Green AI to consider energy consumption as a metric of equal importance to accuracy and to reduce any irrelevant tasks or energy usage. We examine the training stage of the deep learning pipeline from a sustainability perspective, through the study of hyperparameter tuning strategies and the model complexity, two factors vastly impacting the overall pipeline's energy consumption. First, we investigate the effectiveness of grid search, random search and Bayesian optimisation during hyperparameter tuning, and we find that Bayesian optimisation significantly dominates the other strategies. Furthermore, we analyse the architecture of convolutional neural networks with the energy consumption of three prominent layer types: convolutional, linear and ReLU layers. The results show that convolutional layers are the most computationally expensive by a strong margin. Additionally, we observe diminishing returns in accuracy for more energy-hungry models. The overall energy consumption of training can be halved by reducing the network complexity. In conclusion, we highlight innovative and promising energy-efficient practices for training deep learning models. To expand the application of Green AI, we advocate for a shift in the design of deep learning models, by considering the trade-off between energy efficiency and accuracy.
Subject
deep learning
green ai
green software
hyper-parameter tuning
network architecture
To reference this document use:
http://resolver.tudelft.nl/uuid:754e6da6-602b-4c2e-83c7-5a81673ed230
DOI
https://doi.org/10.1109/CAIN58948.2023.00012
Publisher
IEEE
Embargo date
2024-01-04
ISBN
979-8-3503-0113-7
Source
Proceedings - 2023 IEEE/ACM 2nd International Conference on AI Engineering - Software Engineering for AI, CAIN 2023
Event
2nd IEEE/ACM International Conference on AI Engineering - Software Engineering for AI, CAIN 2023, 2023-05-15 → 2023-05-16, Melbourne, Australia
Series
Proceedings - 2023 IEEE/ACM 2nd International Conference on AI Engineering - Software Engineering for AI, CAIN 2023
Bibliographical note
Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public
Part of collection
Institutional Repository
Document type
conference paper
Rights
© 2023 Tim Yarally, Luis Cruz, Daniel Feitosa, J. Sallou, A. van Deursen