Title
S-QUERY: Opening the Black Box of Internal Stream Processor State
Author
Verheijde, Jim (Student TU Delft)
Karakoidas, Vassilios (Delivery Hero SE)
Fragkoulis, M. (TU Delft Web Information Systems) 
Katsifodimos, A (TU Delft Web Information Systems)
Contributor
O'Conner, L. (editor)
Date
2022
Abstract
Distributed streaming dataflow systems have evolved into scalable and fault-tolerant production-grade systems. Their applicability has departed from the mere analysis of streaming windows and complex-event processing, and now includes cloud applications and machine learning inference. Although the advancements in the state management of streaming systems have contributed significantly to their maturity, the internal state of streaming operators has been so far hidden from external applications. However, that internal state can be seen as a materialized view that can be used for analytics, monitoring, and debugging. In this paper we argue that exposing the internal state of streaming systems to outside applications by making it queryable, opens the road for novel use cases. To this end, we introduce S-QUERY: an approach and reference architecture where the state of stream processors can be queried - either live or through snapshots, achieving different isolation levels. We show how this new capability can be implemented in an existing open-source stream processor, and how queryable state can affect the performance of such a system. Our experimental evaluation suggests that the snapshot configuration adds only up to 8ms latency in the 99.99thpercentile and negligible increase in 0-90thpercentiles.
To reference this document use:
http://resolver.tudelft.nl/uuid:464df924-f824-4c27-8607-9152df4c6d41
DOI
https://doi.org/10.1109/ICDE53745.2022.00103
Publisher
IEEE, Piscataway
Embargo date
2023-07-01
ISBN
978-1-6654-0884-4
Source
Proceedings of the 2022 IEEE 38th International Conference on Data Engineering (ICDE)
Event
2022 IEEE 38th International Conference on Data Engineering (ICDE), 2022-05-09 → 2022-05-12, Kuala Lumpur, Malaysia
Series
Proceedings - International Conference on Data Engineering, 1084-4627, 2022-May
Bibliographical note
Green Open Access added to TU Delft Institutional Repository 'You share, we take care!' - Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.
Part of collection
Institutional Repository
Document type
conference paper
Rights
© 2022 Jim Verheijde, Vassilios Karakoidas, M. Fragkoulis, A Katsifodimos