The introduction of cloud hosting has made it possible to elastically provision distributed stream processing systems (SPEs). By dynamically scaling the different operators of the system, resource consumption can be minimised while meeting the system service-level objectives. In the literature, many different auto-scaling techniques are proposed that make scaling decisions based on the current state of the system. However, these techniques are poorly evaluated and are rarely compared with each other. This makes it difficult to determine the state-of-the-art for auto-scaling techniques targeting SPEs, which slows down its development. In this paper, we design and implement a modular framework to evaluate the performance of state-of-the-art auto-scalers targeting SPEs. We implement state-of-the-art auto-scalers Dhalion, DS2, and Varga et al., using Kubernetes horizontal pod auto-scaler as baseline. We perform an end-to-end experimental evaluation of the auto-scalers and investigate their performance when run on different queries and workload patterns. Furthermore, we investigate the convergence time of the auto-scalers and evaluate their scaling accuracy. The results emphasise the difficulty of capturing the complex relationships of different operators and the struggle to balance resource efficiency and the performance of the system. Moreover, it shows the inherent weakness of reactive auto-scalers to react slowly to changing workloads and reveals the importance of considering the current health of the system when issuing scaling actions.

OKademy is a start-up that wants to improve the healthcare in the Netherlands by improving the process by which graduated medical students are being matched to a hospital team. This is needed since there is a lack of surgery
assistants, while graduate students need to wait on average six months to start their work in a hospital. To match quickly and sufficiently, the hard-skills as well as soft-skills of the student need to be known. To achieve this, universities need to provide OKademy with each student’s soft-skills. However, this takes an excessive amount of effort for the universities. To let the universities benefit from this idea, OKademy wants to have a system which keeps track of the soft-skills and matches students in groups for assignments, thus being beneficial for universities and hospitals. To help Okademy achieve its goal we developed a system that can be used by universities to help the students keep track of their soft-skills. The same system can be used to form theoretically well-functioning groups. These groups should be generated using an optimization algorithm that approximates the best groups regarding the soft-skills of students. In this system, students, referred to as ‘members’, and instructors, referred to as ‘hosts’, can register. Members can be assigned into groups and hosts can create or modify groups and courses accordingly. Members can, after registration, fill in their top 10 soft-skills with a corresponding automatized grade. Based on these skills, they will be assigned to groups for a course created by a host. A course will go through multiple phases. When all groups have completed the assignment, a member is able to give feedback to five random soft-skills of their group members and recommend a new one if desired. Based on this feedback, their soft-skill grades will be adjusted. The possible combinations of groups can be extremely large, therefore, it is not feasible to blindly search for the best group formation. Our algorithm will use the idea of genetic algorithms to explore the search space and approximate the best group formation. The the final product consists of a web application in which the multi-objective group formation optimization algorithm is implemented. To work in a structured manner, we used the Scrum method. Meetings with our client and coach were held every week. The system has been tested by functionality tests, user tests and unit tests, ensuring a functional system. OKademy will use this system in collaboration with universities and hospitals to solve the problem.