A Comparison of Auto-scaling Techniques for Distributed Stream Processing

A Comparison of Auto-scaling Techniques for Distributed Stream Processing

Koper,ook geschreven Jansen, Wybe (TU Delft Electrical Engineering, Mathematics and Computer Science; TU Delft Web Information Systems)

Katsifodimos, A (graduation committee)
Spinellis, D. (graduation committee)
Houben, G.J.P.M. (mentor)

Delft University of Technology

Computer Science | Web Information Systems

2022-05-31

As the world continues to embrace cloud computing, more applications are being scaled elastically. Elastic scaling allows applications to add or remove computing resources based on the load experienced by the application. When the load is high more resources are provisioned enabling the application to keep up with the load. When the load is low resources are removed ensuring that no resources are sitting idle. When implemented correctly elastic scaling allows applications to use fewer resources while maintaining application performance. One type of application that can benefit greatly from elastic scaling is a distributed stream processing application. Distributed stream processing applications are suited well for elastic scaling because the data coming through the data stream can be dynamic. This dynamic data stream makes it difficult to provision the right amount of computing power. One way to solve this problem is by using an auto-scaler that elastically scales the stream processing application. In this thesis, we compare different auto-scaling techniques for the distributed stream processing application Apache Flink. We implement a modern version of DS2 using metrics native to Apache Flink. An auto-scaler designed specifically for Apache Flink by Varga et al.. A modified version of the Dhalion, and a simple CPU based Kubernetes Horizontal Pod Auto-scaler (HPA). We compare the auto-scalers on the average number of resources used, the average latency, and the number of re-scale operations. Our results show the importance of a cooldown period between scaling events. The benefits of incorporating metrics from the message queue into the scaling decision, and that throughput based evaluation methods work well for determining by how much to scale.

Distributed Systems
Stream processing
Auto-scaling

http://resolver.tudelft.nl/uuid:0ae72b82-b3af-4afb-a8d8-8040a226f045

Student theses

master thesis

© 2022 Wybe Koper,ook geschreven Jansen