S-QUERY: Opening the Black Box of Internal Stream Processor State
Verheijde, Jim (Student TU Delft)
Karakoidas, Vassilios (Delivery Hero SE)
Fragkoulis, M. (TU Delft Web Information Systems)
Katsifodimos, A (TU Delft Web Information Systems)
O'Conner, L. (editor)
Distributed streaming dataflow systems have evolved into scalable and fault-tolerant production-grade systems. Their applicability has departed from the mere analysis of streaming windows and complex-event processing, and now includes cloud applications and machine learning inference. Although the advancements in the state management of streaming systems have contributed significantly to their maturity, the internal state of streaming operators has been so far hidden from external applications. However, that internal state can be seen as a materialized view that can be used for analytics, monitoring, and debugging. In this paper we argue that exposing the internal state of streaming systems to outside applications by making it queryable, opens the road for novel use cases. To this end, we introduce S-QUERY: an approach and reference architecture where the state of stream processors can be queried - either live or through snapshots, achieving different isolation levels. We show how this new capability can be implemented in an existing open-source stream processor, and how queryable state can affect the performance of such a system. Our experimental evaluation suggests that the snapshot configuration adds only up to 8ms latency in the 99.99thpercentile and negligible increase in 0-90thpercentiles.
To reference this document use:
Proceedings of the 2022 IEEE 38th International Conference on Data Engineering (ICDE)
2022 IEEE 38th International Conference on Data Engineering (ICDE), 2022-05-09 → 2022-05-12, Kuala Lumpur, Malaysia
Proceedings - International Conference on Data Engineering, 1084-4627, 2022-May
Green Open Access added to TU Delft Institutional Repository 'You share, we take care!' - Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.
Part of collection
© 2022 Jim Verheijde, Vassilios Karakoidas, M. Fragkoulis, A Katsifodimos