S-Query: Opening the Black Box of Internal Stream Processor State

Master thesis (2021)

Authors

J.L. Verheijde Electrical Engineering, Mathematics and Computer Science

Contributors

A Katsifodimos Web Information Systems - EEMCS (supervisor 1)
M. Fragkoulis Web Information Systems - EEMCS (supervisor 2)
G.J.P.M. Houben Web Information Systems - EEMCS (supervisor 2)
Jan S. Rellermeyer Data-Intensive Systems - EEMCS (supervisor 2)
Faculty
Electrical Engineering, Mathematics and Computer Science
Copyright: © 2021 Jim Verheijde
More Info
expand_more

Published Date

27-08-2021

Language

English

Reuse Rights

Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.

Faculty

Electrical Engineering, Mathematics and Computer Science

Abstract

At the moment we are witnessing the maturation of distributed streaming dataflow systems whose use-cases have departed from the mere analysis of streaming windows and complex-event processing, as they now extend to cloud applications, workflows and even e-commerce. The state of streaming operators has been so far hidden from external applications. In this thesis it is argued that exposing this internal state to outside applications by making it queryable, opens the road for novel use-cases. To this end, we introduce S-Query: a system and reference architecture where the state of stream operators can be queried - either live or through snapshots, achieving different isolation levels. It is shown how this can be implemented in an existing open-source stream processor and how queryable state can affect the performance of such a system. Our experimental evaluation suggests that snapshot configuration adds only up to 5ms in latency in the 99.99th percentile and no increase in 0-90th percentiles.