Cascade
From Imperative Code to Stateful Dataflows
M. Schutte (TU Delft - Data-Intensive Systems)
Lucas van Mol (Student TU Delft)
G.C. Christodoulou (TU Delft - Data-Intensive Systems)
Asterios Katsifodimos (TU Delft - Data-Intensive Systems)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
Executing applications in the cloud is becoming increasingly popular, primarily developed as microservices containing imperative code. In our previous work, we have made the case that such applications can benefit from using dataflow-based runtimes in a cloud environment. In particular, dataflow-based runtimes offer significant advantages over imperative code, namely, exactly-once processing, transparent message handling, and coarse-grained fault tolerance offered by dataflow systems. However, dataflow programming is not preferred by developers. In this work we bridge this gap, namely, we present our progress towards creating a suitable intermediate representation (IR) that can be used to compile stateful imperative code into dataflows, enabling seamless migration to the cloud. We then present a compiler pipeline prototype that offers two key benefits: i) it enables program optimizations and data parallelism, and ii) it decouples the input program from the target execution environment, while allowing interesting optimizations. Preliminary experiments demonstrate that our IR optimizations speed up the p50 request latency by 267x on average.
help_outline