Print Email Facebook Twitter Network-Aware Locality Scheduling for Distributed Data Operators in Data Centers Title Network-Aware Locality Scheduling for Distributed Data Operators in Data Centers Author Cheng, Long (North China Electric Power University; Chinese Academy of Sciences) Wang, Ying (Chinese Academy of Sciences) Liu, Qingzhi (Wageningen University & Research) Epema, D.H.J. (TU Delft Data-Intensive Systems) Liu, Cheng (Chinese Academy of Sciences) Mao, Ying (Fordham University) Murphy, John (University College Dublin) Date 2021 Abstract Large data centers are currently the mainstream infrastructures for big data processing. As one of the most fundamental tasks in these environments, the efficient execution of distributed data operators (e.g., join and aggregation) are still challenging current data systems, and one of the key performance issues is network communication time. State-of-the-art methods trying to improve that problem focus on either application-layer data locality optimization to reduce network traffic or on network-layer data flow optimization to increase bandwidth utilization. However, the techniques in the two layers are totally independent from each other, and performance gains from a joint optimization perspective have not yet been explored. In this article, we propose a novel approach called NEAL (NEtwork-Aware Locality scheduling) to bridge this gap, and consequently to further reduce communication time for distributed big data operators. We present the detailed design and implementation of NEAL, and our experimental results demonstrate that NEAL always performs better than current approaches for different workloads and network bandwidth configurations. Subject Data localitySDNbig datacoflow schedulingdata centersdistributed operatorsmetaheuristic To reference this document use: http://resolver.tudelft.nl/uuid:955644ad-dc74-41bf-b90e-7097eed6e8a0 DOI https://doi.org/10.1109/TPDS.2021.3053241 ISSN 1045-9219 Source IEEE Transactions on Parallel and Distributed Systems, 32 (6), 1494-1510 Bibliographical note Accepted author manuscript Part of collection Institutional Repository Document type journal article Rights © 2021 Long Cheng, Ying Wang, Qingzhi Liu, D.H.J. Epema, Cheng Liu, Ying Mao, John Murphy Files PDF neal_IEEE_TPDS_20210125.pdf 1012.4 KB Close viewer /islandora/object/uuid:955644ad-dc74-41bf-b90e-7097eed6e8a0/datastream/OBJ/view