Battling the CPU Bottleneck in Apache Parquet to Arrow Conversion Using FPGA

Conference Paper (2021)
Author(s)

Johan Peltenburg (TU Delft - Computer Engineering)

Lars T.J. Van Leeuwen (Student TU Delft)

J.J. Hoozemans (TU Delft - Computer Engineering)

J. Fang (National Innovation Institute of Defense Technology, TU Delft - Computer Engineering)

Zaid Al-Ars (TU Delft - Computer Engineering)

H. Peter Hofstee (TU Delft - Computer Engineering, IBM)

Research Group
Computer Engineering
Copyright
© 2021 J.W. Peltenburg, Lars T.J. Van Leeuwen, J.J. Hoozemans, J. Fang, Z. Al-Ars, H.P. Hofstee
DOI related publication
https://doi.org/10.1109/ICFPT51103.2020.00048
More Info
expand_more
Publication Year
2021
Language
English
Copyright
© 2021 J.W. Peltenburg, Lars T.J. Van Leeuwen, J.J. Hoozemans, J. Fang, Z. Al-Ars, H.P. Hofstee
Research Group
Computer Engineering
Pages (from-to)
281-286
ISBN (print)
978-1-6654-4622-8
ISBN (electronic)
978-1-6654-2302-1
Reuse Rights

Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.

Abstract

In the domain of big data analytics, the bottleneck of converting storage-focused file formats to in-memory data structures has shifted from the bandwidth of storage to the performance of decoding and decompression software. Two widely used formats for big data storage and in-memory data are Apache Parquet and Apache Arrow, respectively. In order to improve the speed at which data can be loaded from disk to memory, we propose an FPGA accelerator design that converts Parquet files to Arrow in-memory data structures. We describe an extensible, publicly available, free and open-source implementation of the proposed converter that supports various Parquet file configurations. The performance of the converter is measured on an AWS EC2 F1 system and on a POWER9 system using the recently released OpenCAPI interface. A single instance of the converter can reach between 6 and 12 GB/s of end-to-end throughput, and shows up to a threefold improvement over the fastest single-thread CPU implementation. It has a low resource utilization (less than 5% for all types of FPGA resources). This allows scaling out the design to match the bandwidth of the coming generation of accelerator interfaces. The proposed design and implementation can be extended to support more of the many possible Parquet file configurations.

Files

License info not available