Adaptive Distributed Streaming Similarity Joins

Adaptive Distributed Streaming Similarity Joins

Siachamis, G. (TU Delft Web Information Systems)
Psarakis, K. (TU Delft Web Information Systems)
Fragkoulis, M. (Delivery Hero SE)
Papapetrou, Odysseas (Eindhoven University of Technology)
van Deursen, A. (TU Delft Software Technology)
Katsifodimos, A (TU Delft Web Information Systems)

Pasin, Marcelo (editor)

Software Technology

2023

How can we perform similarity joins of multi-dimensional streams in a distributed fashion, achieving low latency? Can we adaptively repartition those streams in order to retain high performance under concept drifts? Current approaches to similarity joins are either restricted to single-node deployments or focus on set-similarity joins, failing to cover the ubiquitous case of metric-space similarity joins. In this paper, we propose the first adaptive distributed streaming similarity join approach that gracefully scales with variable velocity and distribution of multi-dimensional data streams. Our approach can adaptively rebalance the load of nodes in the case of concept drifts, allowing for similarity computations in the general metric space. We implement our approach on top of Apache Flink and evaluate its data partitioning and load balancing schemes on a set of synthetic datasets in terms of latency, comparisons ratio, and data duplication ratio

http://resolver.tudelft.nl/uuid:7110874a-a227-4407-a35e-7f78e4b2d8b8

979-8-4007-0122-1

DEBS '23: Proceedings of the 17th ACM International Conference on Distributed and Event-based Systems

17th ACM International Conference on Distributed and Event-based Systems, 2023-06-27 → 2023-06-30, DEBS '23: 17th ACM International Conference on Distributed and Event-based Systems Neuchatel Switzerland June 27 - 30, 2023, Switzerland

Institutional Repository

conference paper

© 2023 G. Siachamis, K. Psarakis, M. Fragkoulis, Odysseas Papapetrou, A. van Deursen, A Katsifodimos