An FPGA-based Snappy Decompressor-Filter

An FPGA-based Snappy Decompressor-Filter

Qiao, Yang (TU Delft Electrical Engineering, Mathematics and Computer Science)

Hofstee, H.P. (mentor)
Fang, J. (graduation committee)

Delft University of Technology

Electrical Engineering | Microelectronics

2018-01-30

New interfaces to interconnect CPUs and accelerators at memory-class bandwidth pose new opportunities and challenges for the design of accelerators. This thesis studies one such accelerator, a decompressor for Parquet files compressed with the Snappy library. Our design targets reconfigurable logic (FPGAs) attached via the open coherent accelerator processor interface(OpenCAPI) at 25.6GB/s. We give an overview of the previous research in hardware-based (de)compression engines and present and analyze our design. Much of the challenge of designing the decompression engine stems from the need to process more than one token per cycle. In our design, a single engine can process two tokens per cycle. A Xilinx KU15P FPGA is expected to support multiple such engines. The input throughput and the output throughput ranges of a single engine are 3.9$\sim$6.3 bytes/cycle and 8.3$\sim$15 bytes/cycle, respectively. Based on the implementation results, a single engine of the proposed design could work at 140MHz, meaning 0.51$\sim$0.82 GB/s input throughput or 1.08$\sim$1.96 GB/s output throughput. The Parquet format enables the parallel decompression of multiple blocks when multiple units are instantiated. With the latest generation of FPGAs, we estimate at most 28 units can be supported leading to a total input/output bandwidth of 14.28/30.24 to 22.96/54.88 GB/s. Because the output bandwidth can exceed the interface bandwidth if multiple engines are supported, the design is especially effective when combined with a filter engine that reduces the output size.

FPGA
Snappy
Parquet
hardware decompression

http://resolver.tudelft.nl/uuid:5ed3a03f-b4ac-4513-8f1c-d7583e7626bc

Student theses

master thesis

© 2018 Yang Qiao