Memory-Disaggregated In-Memory Object Store Framework for Big Data Applications
R. Abrahamse (TU Delft - Electrical Engineering, Mathematics and Computer Science)
Zaid Al-Ars – Mentor (TU Delft - Computer Engineering)
A. Hadnagy – Graduation committee member (TU Delft - Computer Engineering)
BHM Gerritsen – Coach (TU Delft - Computer Science & Engineering-Teaching Team)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
The concept of memory disaggregation has recently been gaining traction in research. With memory disaggregation, data center compute nodes are able to directly access memory on adjacent nodes and can therefore overcome local memory restrictions, introducing a new data management paradigm for distributed computing. This paper proposes and demonstrates a memory disaggregated in-memory object store framework for big data applications by leveraging the newly introduced ThymesisFlow memory disaggregation system. The framework extends the functionality of the existing Apache Arrow Plasma object store framework to distributed systems by enabling clients to easily and efficiently produce and consume data objects across multiple compute nodes. This allows big data applications to increasingly leverage parallel processing at reduced development cost. In addition, the paper includes latency and throughput measurements to evaluate the framework’s performance and to guide the design of future systems that leverage memory disaggregation as well as the newly presented framework.