Print Email Facebook Twitter Online state migration in modern stream processing engines Title Online state migration in modern stream processing engines Author Veneti, Theodoros (TU Delft Electrical Engineering, Mathematics and Computer Science; TU Delft Web Information Systems) Contributor Katsifodimos, A (mentor) Decouchant, Jérémie (graduation committee) Degree granting institution Delft University of Technology Programme Computer Science | Data Science and Technology Date 2023-11-07 Abstract Stream Processing Engines (SPEs) are called upon to help solve problems around big and volatile data, while satisfying the needs for near real-time processing. In order for such systems to be considered effective solutions to such problems at scale, efficient elasticity and non dataflow-disturbing reconfiguration operations within are a necessity. To that end, we visit the problem of online state migration, as the biggest obstacle in achieving such a desired behaviour, in SPEs that support stateful functions. We make an attempt to formally define the problem and associated sub-tasks, compare existing solutions and identify key aspects, as well as design and implement our own solution. Our testing shows that the lazy-fetch online state migration process proposed, outperforms a simple baseline state migration design by orders of magnitude in end-to-end latency observed, scales much better under increased workloads and relies on consistent design concepts to claim exactly-once semantics. Subject SPEstream processing enginesstate migrationonline state migrationfetch on demandreconfiguration To reference this document use: http://resolver.tudelft.nl/uuid:5d602f53-1fb3-41a0-81dd-28ffd0d20787 Part of collection Student theses Document type master thesis Rights © 2023 Theodoros Veneti Files PDF Thesis_report_Theodoros_Venetis.pdf 1.65 MB Close viewer /islandora/object/uuid:5d602f53-1fb3-41a0-81dd-28ffd0d20787/datastream/OBJ/view