Print Email Facebook Twitter Batching for Green AI - An Exploratory Study on Inference Title Batching for Green AI - An Exploratory Study on Inference Author Yarally, T.E.R. Cruz, Luis (TU Delft Software Engineering) Feitosa, Daniel (University Medical Center Groningen) Sallou, J. (TU Delft Software Engineering) van Deursen, A. (TU Delft Software Engineering) Date 2023 Abstract The batch size is an essential parameter to tune during the development of new neural networks. Amongst other quality indicators, it has a large degree of influence on the model’s accuracy, generalisability, training times and parallelisability. This fact is generally known and commonly studied. However, during the application phase of a deep learning model, when the model is utilised by an end-user for inference, we find that there is a disregard for the potential benefits of introducing a batch size. In this study, we examine the effect of input batching on the energy consumption and response times of five fully-trained neural networks for computer vision that were considered state-of-the-art at the time of their publication. The results suggest that batching has a significant effect on both of these metrics. Furthermore, we present a timeline of the energy efficiency and accuracy of neural networks over the past decade. We find that in general, energy consumption rises at a much steeper pace than accuracy and question the necessity of this evolution. Additionally, we highlight one particular network, ShuffleNetV2 (2018), that achieved a competitive performance for its time while maintaining a much lower energy consumption. Nevertheless, we highlight that the results are model dependent. To reference this document use: http://resolver.tudelft.nl/uuid:c807a83e-0062-498d-8ecb-e10328c6547b DOI https://doi.org/10.1109/SEAA60479.2023.00026 Publisher IEEE Embargo date 2024-07-08 ISBN 979-8-3503-4235-2 Source 49th Euromicro Conference on Software Engineering and Advanced Applications (SEAA) Bibliographical note Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public. Part of collection Institutional Repository Document type conference paper Rights © 2023 T.E.R. Yarally, Luis Cruz, Daniel Feitosa, J. Sallou, A. van Deursen Files file embargo until 2024-07-08