Self-adaptive Executors for Big Data Processing

Conference Paper (2019)
Author(s)

Sobhan Omranian Khorasani (TU Delft - Data-Intensive Systems)

Jan S. Rellermeyer (TU Delft - Data-Intensive Systems)

D.H.J. Epema (TU Delft - Data-Intensive Systems)

Research Group
Data-Intensive Systems
Copyright
© 2019 S. Omranian Khorasani, Jan S. Rellermeyer, D.H.J. Epema
DOI related publication
https://doi.org/10.1145/3361525.3361545
More Info
expand_more
Publication Year
2019
Language
English
Copyright
© 2019 S. Omranian Khorasani, Jan S. Rellermeyer, D.H.J. Epema
Research Group
Data-Intensive Systems
Pages (from-to)
176-188
ISBN (print)
978-1-4503-7009-7
ISBN (electronic)
9781450370097
Reuse Rights

Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.

Abstract

The demand for additional performance due to the rapid increase in the size and importance of data-intensive applications has considerably elevated the complexity of computer architecture. In response, systems offer pre-determined behaviors based on heuristics and then expose a large number of configuration parameters for operators to adjust them to their particular infrastructure. Unfortunately, in practice this leads to a substantial manual tuning effort. In this work, we focus on one of the most impactful tuning decisions in big data systems: the number of executor threads. We first show the impact of I/O contention on the runtime of workloads and a simple static solution to reduce the number of threads for I/O-bound phases. We then present a more elaborate solution in the form of self-adaptive executors which are able to continuously monitor the underlying system resources and detect contentions. This enables the executors to tune their thread pool size dynamically at runtime in order to achieve the best performance. Our experimental results show that being adaptive can significantly reduce the execution time especially in I/O intensive applications such as Terasort and PageRank which see a 34% and 54% reduction in runtime.

Files

P176_khorasani.pdf
(pdf | 1.85 Mb)
- Embargo expired in 09-06-2020
License info not available