Hazelcast jet
Low-latency stream processing at the 99.99th percentile
Can Gencer (Hazelcast Inc)
Marko Topolnik (Hazelcast Inc)
Viliam Ďurina (Hazelcast Inc)
Emin Demirci (Hazelcast Inc)
Ensar B. Kahveci (Hazelcast Inc)
Ali Gürbüz (Hazelcast Inc)
Ondřej Lukáš (Hazelcast Inc)
Marios Fragkoulis (TU Delft - Electrical Engineering, Mathematics and Computer Science)
Asterios Katsifodimos (TU Delft - Electrical Engineering, Mathematics and Computer Science)
undefined More Authors (External organisation)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
Jet is an open-source, high-performance, distributed stream processor built at Hazelcast during the last five years. Jet was engineered with millisecond latency on the 99.99th percentile as its primary design goal. Originally Jet’s purpose was to be an execution engine that performs complex business logic on top of streams generated by Hazelcast’s In-memory Data Grid (IMDG): a set of in-memory, partitioned and replicated data structures. With time, Jet evolved into a full-fledged, scale-out stream processor that can handle out-of-order streams and provide exactly-once processing guarantees. Jet’s end-to-end latency lies in the order of milliseconds, and its throughput in the order of millions of events per CPU-core. This paper presents the main design decisions we made in order to maximize the performance per CPU-core, alongside lessons learned, and an empirical performance evaluation.
help_outline