Library
home (Home) TU Delft Repository login Login

Delay-Resistant Geo-Distributed Analytics

Journal Article (2022)
Author(s)

Habib Mostafaei (Eindhoven University of Technology)

Georgios Smaragdakis (Berlin Institute for the Foundations of Learning and Data, Max Planck Institut für Informatik, TU Delft - Electrical Engineering, Mathematics and Computer Science)

Thomas Zinner (Norwegian University of Science and Technology (NTNU))

Anja Feldmann (Max Planck Institut für Informatik)

Research Group
Cyber Security
DOI related publication
https://doi.org/10.1109/TNSM.2022.3192710 Final published version
More Info
expand_more
Publication Year
2022
Language
English
Research Group
Cyber Security
Issue number
4
Volume number
19
Pages (from-to)
4734-4749
Downloads counter
210
Collections
Institutional Repository
Reuse Rights

Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.

Abstract

Big data analytics platforms have played a critical role in the unprecedented success of data-driven applications. However, real-time and streaming data applications, and recent legislation, e.g., GDPR in Europe, have posed constraints on exchanging and analyzing data, especially personal data, across geographic regions. To address such constraints data has to be processed and analyzed in-situ and aggregated results have to be exchanged among the different sites for further processing. This introduces additional network delays due to the geographic distribution of the sites and potentially affecting the performance of analytics platforms that are designed to operate in datacenters with low network delays. In this paper, we show that the three most popular big data analytics systems (Apache Storm, Apache Spark, and Apache Flink) fail to tolerate round-trip times more than 30 milliseconds even when the input data rate is low. The execution time of distributed big data analytics tasks degrades substantially after this threshold, and some of the systems are more sensitive than others. A closer examination and understanding of the design of these systems show that there is no winner in all wide-area settings. However, we show that it is possible to improve the performance of all these popular big data analytics systems significantly amid even transcontinental delays (where inter-node delay is more than 30 milliseconds) and achieve performance comparable to this within a datacenter for the same load.