FPGA Acceleration of Zstd Compression Algorithm

FPGA Acceleration of Zstd Compression Algorithm

Chen, Jianyu (Student TU Delft; Optiver)
Daverveldt, Maurice (Optiver)
Al-Ars, Z. (TU Delft Computer Engineering)

O'Conner, L. (editor)

2021

With the continued increase in the amount of big data generated and stored in various application domains, such as high-frequency trading, compression techniques are becoming ever more important to reduce the requirements on communication bandwidth and storage capacity. Zstandard (Zstd) is emerging as an important compression algorithm for big data sets capable of achieving a good compression ratio but with a higher speed than comparable algorithms. In this paper, we introduce the architecture of a new hardware compression kernel for Zstd that allows the algorithm to be used for real-time compression of big data streams. In addition, we optimize the proposed architecture for the specific use case of streaming high-frequency trading data. The optimized kernel is implemented on a Xilinx Alveo U200 board. Our optimized implementation allows us to fit ten kernel blocks on one board, which is able to achieve a compression throughput of about 8.6GB/s and compression ratio of about 23.6%. The hardware implementation is open source and publicly available at https://github.com/ChenJianyunp/Hardware-Zstd-Compression-Unit.

FPGA
Zstd
compression

http://resolver.tudelft.nl/uuid:4966f075-d467-485a-9e20-54fbfe2bacf2

https://doi.org/10.1109/IPDPSW52791.2021.00035

IEEE, Piscataway

978-1-6654-1192-9

2021 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)

2021 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), 2021-06-17 → 2021-06-21, Virtually at Portland, United States

Accepted author manuscript

Institutional Repository

conference paper

© 2021 Jianyu Chen, Maurice Daverveldt, Z. Al-Ars