Similarity joins are operations which involve identifying similar pairs of records within one or multiple datasets. These operations are typically time-sensitive, as timely identification of relations can lead to increased profitability. Therefore, it is advantageous to analyze them using a stream processing system, which offers real-time capabilities. Due to the computational complexity of comparing numerous records, similarity joins can be resource-intensive.

To address this challenge, employing a distributed setting for executing the operations proves to be the most effective approach for resource management. In this research, we evaluate four distinct distributed systems designed for similarity joins in stream processing environments. The primary objective is to assess their individual strengths and weaknesses, as well as their overall efficiency. Our investigation reveals that certain solutions exhibit superior scalability and resource utilization, while highlighting the potential for further advancements in this domain.

The development of data stream processing has become one of the key themes in the database and distributed system community throughout the world as data has grown on a large scale and in a range of industries over the last several years. Because data stream processing is a relatively new breakthrough in data-driven approaches, several teams at ING are investigating its possibilities. Thus, this thesis aims to provide insight on data stream processing practices at ING using research survey methodology. We conducted an extensive study that included a review of data streaming academic publications, online questionnaire distributed to 45 practitioners at ING, and in-depth interviews with 5 streaming practitioners. Our survey research aimed at understanding: (i) the use cases of data streaming; (ii) the types of streamed data users have; (iii) the streaming tasks and computation users run on their stream; (iv) the machine learning task users performed in their streams; and (v) the streaming software and tools used to process their streams. Results from academic review became the basis of designing the questionnaire. We discussed the answers of the participants to our questionnaire by highlighting common trends and challenges they faced. Through our interviews, we were able to get detailed answers on some of our questions. Our research discovered several interesting observations regarding data stream processing in practice. Particularly, real-time monitoring and event categorization are the popular use case for data streaming, data contained in streams represent a diverse range of entities and is homogeneous in format, type and category, machine learning implementation in streaming environment is prevalence, Apache Kafka is a commonly used stream processing engine and complexity of data streaming implementation is the challenge most expressed by our participants.